{
  "version": 3,
  "sources": ["ssg:https://framerusercontent.com/modules/YVgynkVt96a0ES9Gk8yz/nd9dAuUAU6tvDmu3yCzj/eo4RAmtig-20.js"],
  "sourcesContent": ["import{jsx as e,jsxs as t}from\"react/jsx-runtime\";import{ComponentPresetsConsumer as a,Link as n}from\"framer\";import{motion as i}from\"framer-motion\";import*as o from\"react\";import{Youtube as r}from\"https://framerusercontent.com/modules/NEd4VmDdsxM3StIUbddO/DDzyuYPF56TuI0bfUu2z/YouTube.js\";export const richText=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"h2\",{children:\"A data-driven approach to machine unlearning for generative language models\"}),/*#__PURE__*/e(\"p\",{children:\"In today\u2019s tech landscape, you\u2019d be hard pressed to find someone who hasn\u2019t heard of machine learning. Over the last decade the research field has been so trendy that even those outside the industry are now familiar with terms such as Artificial Intelligence (AI), Neural Networks (NNs), and Machine Learning (ML). \"}),/*#__PURE__*/t(\"p\",{children:[\"However, when it comes to machine unlearning, it seems the legal industry has heard more about it than the tech community. The recent boom in Large Language Models (LLMs), which in the fast-paced world of IT feels like a decade even if it\u2019s only been 1\u20132 years, has unearthed hundreds of unresolved ethical and legal issues related to AI development. Novelists \",/*#__PURE__*/e(n,{href:\"https://www.hollywoodreporter.com/business/business-news/authors-sue-meta-openai-class-action-1235588711/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"are suing\"})}),\" OpenAI for using their texts to train GPT models without consent. Twitter is \",/*#__PURE__*/e(n,{href:\"https://twitter.com/mattdeitke/status/1638608472525897728\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"abuzz with critical comments\"})}),\" from artists who believe their works were used in violation of copyright laws. \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2307.03941.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:'Complying with \"the right to be forgotten\" has become extremely challenging'})}),\".\"]}),/*#__PURE__*/e(\"img\",{alt:\"A data-driven approach to machine unlearning for generative language models\",className:\"framer-image\",height:\"1935\",src:\"https://framerusercontent.com/images/S7rRFTsvAcEaG1XnxTgrUE5yTo.webp\",srcSet:\"https://framerusercontent.com/images/S7rRFTsvAcEaG1XnxTgrUE5yTo.webp?scale-down-to=1024 677w,https://framerusercontent.com/images/S7rRFTsvAcEaG1XnxTgrUE5yTo.webp?scale-down-to=2048 1354w,https://framerusercontent.com/images/S7rRFTsvAcEaG1XnxTgrUE5yTo.webp 2560w\",style:{aspectRatio:\"2560 / 3871\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[\"Much like \",/*#__PURE__*/e(n,{href:\"https://towardsdatascience.com/what-is-ai-alignment-2bbbe4633c7f\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"AI alignment\"})}),\", machine unlearning appears to be an overlooked field, given the limited open-sourced solutions available. I believe that machine unlearning exploration should be encouraged and popularized, especially considering that the current laws and ethical norms surrounding AI usage are underdeveloped and severely lack mechanisms for data protection. In this article, I would like to suggest some practical improvements to \",/*#__PURE__*/e(n,{href:\"https://browse.arxiv.org/pdf/2310.02238.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"one of the first applied unlearning techniques for generative language models\"})}),\".\"]}),/*#__PURE__*/e(\"h2\",{children:\"What is machine unlearning?\"}),/*#__PURE__*/e(\"p\",{children:'The term \"machine unlearning\" or \"machine forgetting\" means exactly what it sounds like: it includes techniques designed to erase requested information from a machine learning model\u2019s \"knowledge storage\". However, it\u2019s far from intuitive when you need to consider actual methods to achieve this efficiently in terms of time, computational resources, and model performance on the \"not unlearned\" data. An obvious solution is to retrain models from scratch using the initial dataset while excluding the \"forget set\" \u2014 but this would be an extremely impractical approach to deep neural network unlearning.'}),/*#__PURE__*/e(\"img\",{alt:'\"Machine Unlearning Framework\" from Survey of Machine Unlearning',className:\"framer-image\",height:\"549\",src:\"https://framerusercontent.com/images/cdMdfMuzuI0eiJIpj2GeBeG12ek.webp\",srcSet:\"https://framerusercontent.com/images/cdMdfMuzuI0eiJIpj2GeBeG12ek.webp?scale-down-to=512 512w,https://framerusercontent.com/images/cdMdfMuzuI0eiJIpj2GeBeG12ek.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/cdMdfMuzuI0eiJIpj2GeBeG12ek.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/cdMdfMuzuI0eiJIpj2GeBeG12ek.webp 2560w\",style:{aspectRatio:\"2560 / 1098\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:'\"Machine Unlearning Framework\" from '}),/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2209.02299.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:/*#__PURE__*/e(\"em\",{children:\"Survey of Machine Unlearning\"})})})]}),/*#__PURE__*/t(\"p\",{children:['The core research findings in the field of machine unlearning are concisely compiled in \"',/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2209.02299.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"A Survey of Machine Unlearning\"})}),'\". Another article that covers the basics with accessible explanations is \"',/*#__PURE__*/e(n,{href:\"https://towardsdatascience.com/machine-unlearning-the-duty-of-forgetting-3666e5b9f6e5\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Machine unlearning: The duty of forgetting\"})}),'\". While I personally recommend these resources, you can find a multitude of other quality research materials on the subject. Yet in terms of practical applications, there remains much to be done.']}),/*#__PURE__*/t(\"p\",{children:[\"A promising initiative that might shift this field from theoretical exploration to practical application is the ongoing \",/*#__PURE__*/e(n,{href:\"https://www.kaggle.com/competitions/neurips-2023-machine-unlearning/overview\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"NeurIPS 2023 Machine Unlearning challenge\"})}),\". Here, participants compete to create an unlearning algorithm for the ResNet18 Convolutional Neural Network.\"]}),/*#__PURE__*/e(\"img\",{alt:\"NeurIPS 2023 Machine Unlearning challenge on Kaggle\",className:\"framer-image\",height:\"827\",src:\"https://framerusercontent.com/images/OAZ0J6wQPKWDAIrGqF0Y5J7aR4.webp\",srcSet:\"https://framerusercontent.com/images/OAZ0J6wQPKWDAIrGqF0Y5J7aR4.webp?scale-down-to=512 512w,https://framerusercontent.com/images/OAZ0J6wQPKWDAIrGqF0Y5J7aR4.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/OAZ0J6wQPKWDAIrGqF0Y5J7aR4.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/OAZ0J6wQPKWDAIrGqF0Y5J7aR4.webp 2560w\",style:{aspectRatio:\"2560 / 1654\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://www.kaggle.com/competitions/neurips-2023-machine-unlearning/overview\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:/*#__PURE__*/e(\"em\",{children:\"NeurIPS 2023 Machine Unlearning challenge\"})})}),/*#__PURE__*/e(\"em\",{children:\" on \"}),/*#__PURE__*/e(n,{href:\"https://www.kaggle.com/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:/*#__PURE__*/e(\"em\",{children:\"Kaggle\"})})})]}),/*#__PURE__*/e(\"h2\",{children:\"Machine unlearning for generative language models\"}),/*#__PURE__*/t(\"p\",{children:[\"Considering the widespread accessibility and promotion of generative language models to the vast majority of internet users, there\u2019s a critical need for unlearning mechanisms. One of the first successful techniques was recently published as open source; you can find the details in \u201C\",/*#__PURE__*/e(n,{href:\"https://browse.arxiv.org/pdf/2310.02238.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Who\u2019s Harry Potter? Approximate Unlearning in LLMs\"})}),\"\u201D by Ronen Eldan and Mark Russinovich.\"]}),/*#__PURE__*/e(\"img\",{alt:\"Image generated with StableDiffusion\",className:\"framer-image\",height:\"1280\",src:\"https://framerusercontent.com/images/sUM9DoFOtgek0ferUv9kJukE.webp\",srcSet:\"https://framerusercontent.com/images/sUM9DoFOtgek0ferUv9kJukE.webp?scale-down-to=512 512w,https://framerusercontent.com/images/sUM9DoFOtgek0ferUv9kJukE.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/sUM9DoFOtgek0ferUv9kJukE.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/sUM9DoFOtgek0ferUv9kJukE.webp 2560w\",style:{aspectRatio:\"2560 / 2560\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Image generated with \"}),/*#__PURE__*/e(n,{href:\"https://stablediffusionweb.com/#demo\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:/*#__PURE__*/e(\"em\",{children:\"StableDiffusion\"})})})]}),/*#__PURE__*/t(\"p\",{children:[\"The authors use a data augmentation approach for machine unlearning on the \",/*#__PURE__*/e(n,{href:\"https://huggingface.co/meta-llama/Llama-2-7b-chat-hf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Llama 2 7b chat model\"})}),\" released this summer by Meta. The chosen unlearning target, also known as the \u201Cforget set\u201D, is the Harry Potter saga (ingenious, these muggles!), which is a perfect example of machine unlearning due to the possible violation of copyright law. They show that with just one GPU hour of fine-tuning, the resulting model is unable to recall most of the Harry Potter-related content, while its performance on common benchmarks remains almost unaffected.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Overview of the data augmentation approach\"}),/*#__PURE__*/t(\"p\",{children:['The main goal of the approach is to make Llama 2 7b forget the linkage between entities from a defined forget set (\"Harry\" ',/*#__PURE__*/e(\"code\",{children:\"<is friends with>\"}),' \"Hermione\") by giving the model plausible generic alternatives (\"Harry\" ',/*#__PURE__*/e(\"code\",{children:\"<is friends with>\"}),' \"Sally\"). To provide these alternatives as target labels in a ',/*#__PURE__*/e(\"strong\",{children:\"fine-tuning dataset\"}),', idiosyncratic terms from the \"domain to be forgotten\" should be highly penalized during the generation of targets. Such penalization could be achieved by combining in equation (1) logits generated by a ',/*#__PURE__*/e(\"strong\",{children:\"reinforced model\"}),\" on the original input\u200A \u2014\u200A Harry Potter books\u200A \u2014\u200A and by a \",/*#__PURE__*/e(\"strong\",{children:\"baseline model\"}),\" on a \",/*#__PURE__*/e(\"strong\",{children:\"generic translation\"}),\" of the original input.\"]}),/*#__PURE__*/e(\"img\",{alt:'Equation (1) from \"Who\u2019s Harry Potter? Approximate Unlearning in LLMs\"',className:\"framer-image\",height:\"112\",src:\"https://framerusercontent.com/images/SNKxZf5ELM9gMrOd45kzeZeKk.webp\",srcSet:\"https://framerusercontent.com/images/SNKxZf5ELM9gMrOd45kzeZeKk.webp?scale-down-to=512 512w,https://framerusercontent.com/images/SNKxZf5ELM9gMrOd45kzeZeKk.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/SNKxZf5ELM9gMrOd45kzeZeKk.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/SNKxZf5ELM9gMrOd45kzeZeKk.webp 2560w\",style:{aspectRatio:\"2560 / 224\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Equation (1) from \"}),/*#__PURE__*/e(n,{href:\"https://browse.arxiv.org/pdf/2310.02238.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:/*#__PURE__*/e(\"em\",{children:'\"Who\u2019s Harry Potter? Approximate Unlearning in LLMs\"'})})})]}),/*#__PURE__*/t(\"p\",{children:[\"The \",/*#__PURE__*/e(\"strong\",{children:\"reinforced model\"}),\" is Llama 2 7b fine-tuned additionally on Harry Potter novels. The \",/*#__PURE__*/e(\"strong\",{children:\"baseline model\"}),\" is untuned Llama 2 7b. To shift the \",/*#__PURE__*/e(\"strong\",{children:\"baseline model\"}),\"\u2019s output distribution away from the Harry Potter theme, the authors replace idiosyncratic terms in the original input with generic ones so the model generates a next word based on a context unrelated to the Harry Potter saga. To automate such replacements, the authors introduce a dictionary of \",/*#__PURE__*/e(\"strong\",{children:\"anchor terms\"}),\" \u200A\u2014 \u200Aterms specific to \u201CHarry Potter\u201D\u200A\u2014\u200A mapped onto \",/*#__PURE__*/e(\"strong\",{children:\"generic translations\"}),\". The dictionary is fully gathered by \",/*#__PURE__*/e(n,{href:\"https://openai.com/research/gpt-4\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"GPT-4\"})}),\".\"]}),/*#__PURE__*/e(\"img\",{alt:\"'Anchor Terms': 'Generic translations' from \\\"Who\u2019s Harry Potter? Approximate Unlearning in LLMs\\\"\",className:\"framer-image\",height:\"559\",src:\"https://framerusercontent.com/images/8QRux3kg5jkzjEpuWf5aTZ7Dlg.webp\",srcSet:\"https://framerusercontent.com/images/8QRux3kg5jkzjEpuWf5aTZ7Dlg.webp?scale-down-to=512 512w,https://framerusercontent.com/images/8QRux3kg5jkzjEpuWf5aTZ7Dlg.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/8QRux3kg5jkzjEpuWf5aTZ7Dlg.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/8QRux3kg5jkzjEpuWf5aTZ7Dlg.webp 2560w\",style:{aspectRatio:\"2560 / 1118\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"'Anchor Terms': 'Generic translations' from \"}),/*#__PURE__*/e(n,{href:\"https://browse.arxiv.org/pdf/2310.02238.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:/*#__PURE__*/e(\"em\",{children:'\"Who\u2019s Harry Potter? Approximate Unlearning in LLMs\"'})})})]}),/*#__PURE__*/t(\"p\",{children:[\"The resulting \",/*#__PURE__*/e(\"strong\",{children:\"fine-tuning dataset\"}),\" consists of \u200Atokenized blocks of text from Harry Potter books in a one-to-one mapping to target labels\u200A, which are \u200Atokens corresponding to the maximal entries of the \",/*#__PURE__*/e(\"em\",{children:\"v_generic\"}),\" from the equation (1).\"]}),/*#__PURE__*/e(\"img\",{alt:'A sample of the fine-tuning dataset from \"Who\u2019s Harry Potter? Approximate Unlearning in LLMs\"',className:\"framer-image\",height:\"759\",src:\"https://framerusercontent.com/images/FbTGLsOFM0bPY7JnxPbFLohK2Qg.webp\",srcSet:\"https://framerusercontent.com/images/FbTGLsOFM0bPY7JnxPbFLohK2Qg.webp?scale-down-to=512 512w,https://framerusercontent.com/images/FbTGLsOFM0bPY7JnxPbFLohK2Qg.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/FbTGLsOFM0bPY7JnxPbFLohK2Qg.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/FbTGLsOFM0bPY7JnxPbFLohK2Qg.webp 2560w\",style:{aspectRatio:\"2560 / 1518\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"A sample of the fine-tuning dataset from \"}),/*#__PURE__*/e(n,{href:\"https://browse.arxiv.org/pdf/2310.02238.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:/*#__PURE__*/e(\"em\",{children:'\"Who\u2019s Harry Potter? Approximate Unlearning in LLMs\"'})})})]}),/*#__PURE__*/e(\"p\",{children:\"To summarize, the authors describe four steps in the unlearning process:\"}),/*#__PURE__*/e(\"img\",{alt:'Machine Unlearning Algorithm from \"Who\u2019s Harry Potter? Approximate Unlearning in LLMs\"',className:\"framer-image\",height:\"408\",src:\"https://framerusercontent.com/images/SjwyWuMmgiImPGsSMFCgphPR6Ww.webp\",srcSet:\"https://framerusercontent.com/images/SjwyWuMmgiImPGsSMFCgphPR6Ww.webp?scale-down-to=512 512w,https://framerusercontent.com/images/SjwyWuMmgiImPGsSMFCgphPR6Ww.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/SjwyWuMmgiImPGsSMFCgphPR6Ww.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/SjwyWuMmgiImPGsSMFCgphPR6Ww.webp 2560w\",style:{aspectRatio:\"2560 / 816\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Machine Unlearning Algorithm from \"}),/*#__PURE__*/e(n,{href:\"https://browse.arxiv.org/pdf/2310.02238.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:/*#__PURE__*/e(\"em\",{children:'\"Who\u2019s Harry Potter? Approximate Unlearning in LLMs\"'})})})]}),/*#__PURE__*/e(\"h2\",{children:\"Leveraging the approach: key challenges\"}),/*#__PURE__*/e(\"p\",{children:\"The results of the data augmentation approach are promising, encouraging further application in similar tasks. Yet, the authors left some room for improvement in several application stages.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Dependency on GPT-4\u2019s existing knowledge:\"}),\" The algorithm to some extent depends on GPT-4\u2019s prior understanding of the Harry Potter series to generate generic translations. While the model is expected to have extensive knowledge of the Harry Potter realm, a reassessment by fans of the series could provide invaluable insights.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Challenges with idiosyncratic terms:\"}),\" Penalizing all unique terms related to the series poses an issue. For instance, replacing every instance of 'Harry' with a common name like 'John' disrupts the model's grasp of natural language, leading to sentences like, \\\"Harry went up to him and said, 'Hi, my name is \",/*#__PURE__*/e(\"em\",{children:\"John\"}),\"'.\\\" To address this, the authors employ the following strategy:\"]}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Excluding repeated instances of anchored terms from contributing to the loss function beyond their initial occurrence.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Lowering the likelihood of logits connected to translations of terms that have appeared before.\"})})]}),/*#__PURE__*/t(\"p\",{children:[\"However, this strategy also affects the model\u2019s general language comprehension. A plausible alternative useful for the fine-tuning dataset would be, for example, \\\"Harry went up to him and said, 'Hi, my name is \",/*#__PURE__*/e(\"em\",{children:\"Harold\"}),\"'.\\\"\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Evaluation techniques:\"}),\" The team utilized GPT-4 for an initial evaluation, comprising 300 Harry Potter prompt completions, and further analysis of completions. Nonetheless, they acknowledged its limitations in accuracy, opting for manual inspections of the results for more thorough verification in their final training. The authors have not provided insights on how to set up such a manual inspection.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Overcoming the challenges\"}),/*#__PURE__*/e(\"p\",{children:\"A more effective way to address the key challenges would be a hybrid approach that combines human insight and Large Language Models (LLMs).\"}),/*#__PURE__*/e(\"p\",{children:\"In order to harness the collective strengths of human intuition and large language models, I have designed three Toloka project interfaces that facilitate collaborative labeling using LLMs and the crowd. Each interface designed for human labeling is tailored to a challenge listed above.\"}),/*#__PURE__*/e(\"h3\",{children:\"Project 1. Dependency on GPT-4\u2019s existing knowledge\"}),/*#__PURE__*/e(\"img\",{alt:\"Image by author\",className:\"framer-image\",height:\"2241\",src:\"https://framerusercontent.com/images/7lWfjgHygtK56FXCOvsHPdXd0.webp\",srcSet:\"https://framerusercontent.com/images/7lWfjgHygtK56FXCOvsHPdXd0.webp?scale-down-to=1024 584w,https://framerusercontent.com/images/7lWfjgHygtK56FXCOvsHPdXd0.webp?scale-down-to=2048 1169w,https://framerusercontent.com/images/7lWfjgHygtK56FXCOvsHPdXd0.webp?scale-down-to=4096 2339w,https://framerusercontent.com/images/7lWfjgHygtK56FXCOvsHPdXd0.webp 2560w\",style:{aspectRatio:\"2560 / 4482\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Image by author\"})}),/*#__PURE__*/t(\"p\",{children:[\"Use the \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/docs/template-builder/templates/named-entity-recognition/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Named Entity Recognition (NER) Project Template\"})}),\" to correct GPT-4 NER choices for a dictionary of anchor terms. As input, provide the text and GPT-4\u2019s selection of terms (you can ask the model to return positions in the text directly), and instruct the crowd to correct and complement the selected entities.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Project 2. Challenges with idiosyncratic terms\"}),/*#__PURE__*/e(\"img\",{alt:\"Image by author 2\",className:\"framer-image\",height:\"2281\",src:\"https://framerusercontent.com/images/e2hHrDaqGZ6KkldhEFXyPaos4.webp\",srcSet:\"https://framerusercontent.com/images/e2hHrDaqGZ6KkldhEFXyPaos4.webp?scale-down-to=1024 574w,https://framerusercontent.com/images/e2hHrDaqGZ6KkldhEFXyPaos4.webp?scale-down-to=2048 1148w,https://framerusercontent.com/images/e2hHrDaqGZ6KkldhEFXyPaos4.webp?scale-down-to=4096 2297w,https://framerusercontent.com/images/e2hHrDaqGZ6KkldhEFXyPaos4.webp 2560w\",style:{aspectRatio:\"2560 / 4563\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Image by author\"})}),/*#__PURE__*/t(\"p\",{children:[\"With the help of a baseline model, check on linguistic correctness prompts with completions done by the baseline model on a generic translation of the original input. All examples where the baseline model is unsure of an answer (the probability of output tokens is below a certain threshold, chosen by you empirically) should be sent to a crowdsourcing project with the interface shown on the image. You can easily create it using Toloka\u2019s \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/docs/template-builder/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"template builder\"})}),\".\"]}),/*#__PURE__*/e(\"h3\",{children:\"Project 3. Evaluation techniques\"}),/*#__PURE__*/e(\"img\",{alt:\"Image by author 3\",className:\"framer-image\",height:\"2299\",src:\"https://framerusercontent.com/images/L46tGKyBuiHtlWHndUZsgpFN4XA.webp\",srcSet:\"https://framerusercontent.com/images/L46tGKyBuiHtlWHndUZsgpFN4XA.webp?scale-down-to=1024 570w,https://framerusercontent.com/images/L46tGKyBuiHtlWHndUZsgpFN4XA.webp?scale-down-to=2048 1140w,https://framerusercontent.com/images/L46tGKyBuiHtlWHndUZsgpFN4XA.webp?scale-down-to=4096 2280w,https://framerusercontent.com/images/L46tGKyBuiHtlWHndUZsgpFN4XA.webp 2560w\",style:{aspectRatio:\"2560 / 4598\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Image by author\"})}),/*#__PURE__*/t(\"p\",{children:[\"Manual inspection of the evaluation done by GPT-4 can be designed like in the image above.This is simple to set up using the \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/docs/template-builder/templates/text-classification/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"text classification template\"})}),\" in Toloka.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/t(\"p\",{children:[\"At Toloka, we've implemented various data-driven approaches to both machine learning and unlearning. For instance, we secured \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/blog/bigcode-project/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Personal Identifiable Information (PII)\"})}),\" in Large Language Models assisting the \",/*#__PURE__*/e(n,{href:\"https://www.bigcode-project.org/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Big Code project\"})}),\" \u2014 and we're ready to power your next AI product.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Whether you\u2019re building your own LLM or need access to high quality data, we\u2019re here to help. Reach out or learn more by visiting: \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/llm-platform/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"LLM-powered zero-code platform for text classification\"})}),\" or \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/large-language-models/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Data labeling for Generative AI and LLM\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Together, we can advance the field of machine unlearning!\"})]});export const richText1=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"There\u2019s a recent trend our researchers couldn\u2019t resist: \",/*#__PURE__*/e(n,{href:\"https://www.chess.com/blog/Black__Knight/chatgpt-chess\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"playing chess with ChatGPT\"})}),\", just for fun. But because we\u2019re Toloka, we added a twist and let a crowd of enthusiastic Tolokers play collectively for the human side. We didn\u2019t expect to see sophisticated moves, but the game held some surprises. \"]}),/*#__PURE__*/e(\"p\",{children:\"In this post we\u2019ll share the setup and gameplay. See if you can guess who won the match!\"}),/*#__PURE__*/e(\"h2\",{children:\"Player profiles: Toloka and GPT 3.5\"}),/*#__PURE__*/e(\"p\",{children:\"The goal of the match was not to discover the best chess players in Toloka\u2019s huge global crowd. We were simply curious how the crowd would play collaboratively and how ChatGPT would fare against them. We wanted to involve as many people as possible, and our only requirement was that participants know how to play the game and follow our guidelines.\"}),/*#__PURE__*/t(\"p\",{children:[\"As for ChatGPT, it\u2019s not trained to play chess and it makes a lot of mistakes. But it plays \",/*#__PURE__*/e(n,{href:\"https://medium.com/@ivanreznikov/how-good-is-chatgpt-at-playing-chess-spoiler-youll-be-impressed-35b2d3ac024a\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"surprisingly well for a language model\"})}),\".\"]}),/*#__PURE__*/e(\"p\",{children:\"Since neither the Tolokers nor GPT 3.5 specialize in chess, it seemed like the teams were evenly matched (despite the imbalance of 160 players against one language model).\"}),/*#__PURE__*/e(\"p\",{children:\"GPT 3.5 tends to play strong for the first 15 to 20 moves and then degrade as the game goes on. So the interesting question was whether Tolokers could withstand the pressure in the first phase to hold out for a win.\"}),/*#__PURE__*/e(\"h2\",{children:\"Game setup\"}),/*#__PURE__*/e(\"p\",{children:\"The players analyzed a virtual chessboard in a Toloka task and input the next move in text format, following specific guidelines. We let multiple people answer until the same move was submitted by two or more players, which indicated a consensus and made the move official. There wasn\u2019t any discussion of the game between Tolokers.\"}),/*#__PURE__*/e(\"img\",{alt:\"game setup\",className:\"framer-image\",height:\"574\",src:\"https://framerusercontent.com/images/HhqcadQqWfxW0BeSB0jYUT3oM.webp\",srcSet:\"https://framerusercontent.com/images/HhqcadQqWfxW0BeSB0jYUT3oM.webp?scale-down-to=512 512w,https://framerusercontent.com/images/HhqcadQqWfxW0BeSB0jYUT3oM.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/HhqcadQqWfxW0BeSB0jYUT3oM.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/HhqcadQqWfxW0BeSB0jYUT3oM.webp 2560w\",style:{aspectRatio:\"2560 / 1148\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:\"We used APIs to get moves from ChatGPT and send them to the next Toloka task. Game play was tracked by a chess engine to ensure that all moves were legitimate and to decide when the game was over.\"}),/*#__PURE__*/e(\"p\",{children:\"In total, 1094 Toloka tasks were initiated, with 45% expiring due to the set time limits. Out of the 604 submitted moves, 73% were rejected as invalid. Ultimately, 160 Tolokers successfully participated in the game, resulting in a 37-move chess match.\"}),/*#__PURE__*/e(\"h3\",{children:\"Stats\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Expired\"}),\": 45% of 1094 started tasks\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Rejected\"}),\": 73% of 604 submitted\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Accepted\"}),\": 163 answers from 160 tolokers\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Price\"}),\": $0.04 per move\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cost of the match\"}),\": $9\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"What about cheating and random answers?\"}),/*#__PURE__*/e(\"p\",{children:\"In terms of quality control, we opted for a no-training, no-exam approach. Anyone could try playing, but they were only paid for valid moves that were verified by the chess engine. If they submitted an invalid move, they were banned from the game.\"}),/*#__PURE__*/e(\"p\",{children:\"We wanted to include as many participants as possible, so we placed a 2-hour temporary ban on a player after each valid submission. This helped us avoid having a handful of skilled chess players dominate the decision-making process.\"}),/*#__PURE__*/e(\"p\",{children:\"To ensure fairness and prevent cheating with external chess apps, we imposed a strict time limit for each answer. This gave players just enough time to come up with a move, but not enough time to consult an external source.\"}),/*#__PURE__*/e(\"p\",{children:\"We couldn\u2019t do anything to keep our opponent from trying to cheat. But whenever the model made an illegal move, we asked it to fix it. If it couldn\u2019t come up with a valid move, we offered a list of all the possible moves and asked the model to pick one.\"}),/*#__PURE__*/e(\"h2\",{children:\"The play by play\"}),/*#__PURE__*/e(\"p\",{children:\"Tolokers played white, and GPT played black. Here are some of the highlights of the 37-move chess match that ensued.\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Opening phase. The game begins with Tolokers making unconventional first moves, opting for advancement on the sidelines of the board instead of typical center control.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Move 9. Too many options for Tolokers to choose from \u2014 it takes a long time to make a decision, with about 60 submissions before reaching agreement.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Move 10. GPT displays its first sign of weakness with an irrational move, and Tolokers respond reasonably well. Tolokers are trailing.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Move 14. GPT makes an obvious mistake, leading to a heated exchange of pieces and a check. Tolokers manage to regain balance on the board.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Move 22. In a pivotal moment, GPT overlooks the impending capture of its rook, resulting in the loss of a critical piece.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Endgame. With remarkable precision and elegance, Tolokers strategically turn a pawn into a queen, leading to a series of checkmates and eventual victory.  \"})})]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{\"--aspect-ratio\":\"560 / 315\",aspectRatio:\"560 / 315\",height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(a,{componentIdentifier:\"module:NEd4VmDdsxM3StIUbddO/DDzyuYPF56TuI0bfUu2z/YouTube.js:Youtube\",children:t=>/*#__PURE__*/e(r,{...t,play:\"Off\",shouldMute:!0,thumbnail:\"High Quality\",url:\"https://youtu.be/RX9dwGqt0uk\"})})}),/*#__PURE__*/e(\"p\",{children:\"What\u2019s fascinating about the gameplay is that it feels like intentional teamwork, even though it was a collective effort without any communication between players whatsoever.\"}),/*#__PURE__*/e(\"p\",{children:\"As expected, GPT 3.5 showed strength in the early stages of the game, but Tolokers persevered and ultimately emerged victorious.\"}),/*#__PURE__*/e(\"h2\",{children:\"What we\u2019ve learned\"}),/*#__PURE__*/e(\"p\",{children:\"We came away with a few thoughts on humans and AI that aren\u2019t exactly new, but still intriguing.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Collective intelligence is powerful\"}),\". This was a fun way to explore distributed decision-making with a large (and diverse) group of people. The right setup and quality control helped us harness the wisdom of the crowd for complex problem-solving.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Human creativity and adaptability are still relevant\"}),\". The experiment highlighted the human ability to think creatively and adapt to dynamic situations. Tolokers dominated the endgame with strategic moves that reflected human intuition and the capacity to handle unexpected scenarios with finesse.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"GPT 3.5 is not a chess engine\"}),\". Large language models are not designed to play chess, and they\u2019re not particularly good at it. If you\u2019re serious about playing against AI, you\u2019re better off trying a chess engine like \",/*#__PURE__*/e(n,{href:\"https://stockfishchess.org/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Stockfish\"})}),\". That being said, your first match with a language model could be a memorable experience.\"]}),/*#__PURE__*/t(\"p\",{children:[\"For better or worse, we didn\u2019t prove anything about Tolokers\u2019 mastery of chess. But we witnessed the beauty of solving complex challenges with small contributions from a diverse \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/blog/global-survey-2023/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"group of people\"})}),\" \u2014 the heart of Toloka.\"]})]});export const richText2=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"h2\",{children:\"About the client\"}),/*#__PURE__*/e(\"p\",{children:\"Our client is a browser developer and part of a leading global technology corporation. The browser\u2019s popular Coupons feature helps users find promo codes for thousands of stores to save money while shopping online.\"}),/*#__PURE__*/e(\"p\",{children:\"The Coupons project was designed to grow user loyalty and increase market share for the browser. The initial goal was to help consumers save $1+ billion on shopping, encouraging them to use the browser for all their shopping needs in the future. \"}),/*#__PURE__*/e(\"h2\",{children:\"Client\u2019s challenge\"}),/*#__PURE__*/e(\"p\",{children:\"To reach their goal of saving consumers money, the browser developers needed a reliable workforce to extract promo codes from thousands of websites and verify them for accuracy and validity. Websites are checked daily, as new promo codes are added and old ones expire \u2014 requiring continual manual effort on a large scale.\"}),/*#__PURE__*/e(\"p\",{children:\"While the client had previously employed multiple vendors to source coupons, they were unable to provide coverage for all countries and domains. Lacking the bandwidth to enter new markets with the Coupons feature, the client\u2019s biggest challenge was expanding coupon coverage to countries they had not operated in\u200C, including Germany, France, Italy, Poland, Indonesia, Australia, and South Africa.\"}),/*#__PURE__*/e(\"h2\",{children:\"Toloka\u2019s solution\"}),/*#__PURE__*/e(\"p\",{children:\"Toloka has the infrastructure, expertise, and global crowd to drastically scale the Coupons project. The browser team uses Toloka to cover new markets, with Tolokers (data labelers in the Toloka crowd) extracting and verifying the codes from thousands of sites. Toloka fully manages the data pipeline and quality control to continually deliver the promo codes, eliminating the need for the client to allocate internal resources to the process.\"}),/*#__PURE__*/e(\"p\",{children:\"The Coupons project has 2 stages:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Coupon collection: Tolokers check websites to collect coupons and take screenshots of the promo codes.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Coupon verification: Tolokers re-check the collected promo codes to make sure they are valid and money-saving. \"})})]}),/*#__PURE__*/e(\"p\",{children:\"The Toloka crowd collects and verifies 45,000 promo codes every month from a continually expanding list of markets and domains around the world.\"}),/*#__PURE__*/e(\"h2\",{children:\"Results achieved\"}),/*#__PURE__*/e(\"p\",{children:\"Toloka provides the client with access to previously untapped markets, covering over 22 new regions. New domains and new markets are continually added as part of the plans to expand the Coupons feature.\"}),/*#__PURE__*/e(\"p\",{children:\"After the project\u2019s initiation in 2021, the client successfully reached the goal of creating over $1 billion in savings for their customers by 2022. The project continues to grow with Toloka collecting more coupons than ever before, helping the client consistently achieve and surpass their monthly targets for consumer savings.\"}),/*#__PURE__*/e(\"p\",{children:\"The success of the Coupons feature is a key driver in gaining an increase in market share for the browser. As more consumers continue to rely on it for saving money, the client has seen growth in the browser\u2019s DAU, one of their most significant metrics.\"}),/*#__PURE__*/e(\"h2\",{children:\"The impact\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Client surpassed KPI targets for saving consumers money\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"22 new global markets covered by Toloka\u2019s crowd, and growing \"})})]})]});export const richText3=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Data labeling, which is fundamental for building artificial intelligence systems, has often been an intricate, tedious, and physically demanding undertaking. AI-assisted data labeling is a form of automated data labeling, representing a game-changing approach that utilizes the power of artificial intelligence to accelerate the crucial step of data labeling in machine learning. \"}),/*#__PURE__*/e(\"p\",{children:\"AI-assisted data labeling promises to reshape how we annotate vast datasets, making the procedure swifter and more productive. Although, to be able to show its full potential it still needs guidance. In this article, we will explore what AI-assisted data labeling is, its benefits, challenges, the necessity of human input when working with such AI labeling tools, and its role in shaping the future of AI.\"}),/*#__PURE__*/e(\"h2\",{children:\"The Essence of Data Labeling\"}),/*#__PURE__*/e(\"p\",{children:\"Before we delve into the intricacies of AI-assisted data labeling, let's first understand why data labeling is so essential. In supervised machine learning, models are trained on labeled datasets to make predictions or classifications. These labels serve as the ground truth, enabling algorithms to learn and generalize from the provided examples. Whether it's image segmentation, sentiment analysis in text, or predicting stock prices, labeled data is the core of AI.\"}),/*#__PURE__*/e(\"h2\",{children:\"Manual Data Labeling\"}),/*#__PURE__*/e(\"p\",{children:\"Traditionally, data labeling, also known as annotation, has been a manual endeavor, where human annotators carefully examine and label data points one by one. The primary goal of the annotation process is to create labeled datasets that can be later employed to train machine learning models, particularly supervised learning models. Such data is indispensable for training AI algorithms to recognize and make predictions about specific patterns or objects within data.\"}),/*#__PURE__*/e(\"p\",{children:\"Manual labels are generated by individuals or teams of annotators who review, analyze data, and then apply labels to it based on predefined guidelines or criteria. In some cases, manual labeling tasks require domain expertise. Annotators with subject matter knowledge are better equipped to handle specialized labeling tasks.\"}),/*#__PURE__*/e(\"h3\",{children:\"Data Labeling Tasks\"}),/*#__PURE__*/e(\"p\",{children:\"Manual labeling is used in a wide range of data labeling tasks, including but not limited to:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Text classification. Assigning categories or tags to text documents;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis. Labeling text data with sentiments such as positive, negative, or neutral;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Image annotation. Adding bounding boxes, object labels, or key points to raw images;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Named entity recognition (NER). Identifying and labeling named entities in text, such as names of people, organizations, and locations;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Speech recognition. Transcribing spoken language into written text. Manual data labeling is effective, although compared to AI automation it is slow, expensive, and prone to human errors and biases. As the scale of data and the complexity of AI applications grow, it becomes increasingly challenging to rely solely on manual labeling.\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"AI-Assisted Automated Data Labeling\"}),/*#__PURE__*/e(\"p\",{children:\"AI-assisted data labeling is a shift in the automated labeling paradigm. It combines the strengths of human judgment and AI automation to label data competently and accurately. This automated labeling approach involves machine learning algorithms to help humans label or annotate large datasets more efficiently and accurately.\"}),/*#__PURE__*/e(\"p\",{children:\"Automated labeling typically encompasses two main approaches: one where an AI system autonomously labels raw data, and the other where AI is integrated into annotation applications to expedite and enhance manual labeling procedures. Here by AI-assisted labeling we mean AI systems that employ automated annotation.\"}),/*#__PURE__*/e(\"p\",{children:\"In that approach, machines can precisely label the most basic components after their training is complete. Manual labeling is only required for challenging issues. And it still will be less time-consuming than fully manual labeling.\"}),/*#__PURE__*/e(\"p\",{children:\"While AI-assisted data labeling can automate and streamline many aspects of the annotation process, it is typically most effective when used as a tool to augment people's capabilities rather than replace them entirely. The ideal approach involves a combination of AI and human effort, with AI handling repetitive and straightforward tasks while humans provide expertise, context, and oversight.\"}),/*#__PURE__*/e(\"h2\",{children:\"Benefits of AI-Assisted Data Labeling\"}),/*#__PURE__*/e(\"p\",{children:\"Here are some key advantages of AI-assisted data labeling:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Scalability and complex data processing\"}),\". AI-assisted labeling solutions can easily scale to handle large and diverse datasets. Whether it's thousands or millions of data points, AI can manage the workload efficiently. AI algorithms also can handle complex data types, such as images, audio, and text, making them suitable for a wide range of AI applications across various industries;\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Speed and Efficiency\"}),\". AI-assisted data labeling significantly accelerates the labeling process. With the assistance of AI algorithms, large sets of data can be labeled much faster than relying solely on manual labor;\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cost Reduction\"}),\". By automating parts of the labeling process, organizations can reduce the overall cost of data preparation. This is especially important for businesses looking to manage their budgets effectively while still employing the power of AI;\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Expertise Augmentation\"}),\". Annotators still play a crucial role in the labeling process by verifying AI-generated labels. AI doesn't replace humans but complements their skills, making the overall process more accurate and efficient;\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Resource Allocation\"}),\". By automating routine labeling tasks, organizations can allocate their human resources to more complex and creative aspects of data analysis and model development, maximizing the value of their workforce;\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Enhanced Productivity\"}),\". AI-assisted labeling allows machine learning engineers to focus on designing and fine-tuning machine learning model rather than spending excessive time on manual data preparation.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Is It Possible to Fully Eliminate Human Input While Doing AI-Assisted Labeling?\"}),/*#__PURE__*/e(\"p\",{children:\"AI-assisted data labeling can significantly reduce the amount of human input required in the traditional manual labeling process, but it is challenging to fully eliminate human involvement, particularly in sophisticated and nuanced tasks. Such assignments often involve subtle sophisticated elements that may be difficult for AI systems to handle accurately. Human judgment and domain expertise are essential for handling ambiguity, understanding context, and making decisions in such cases.\"}),/*#__PURE__*/e(\"h3\",{children:\"Human Involvement in Labeling Pipeline\"}),/*#__PURE__*/e(\"p\",{children:\"That\u2019s what humans can do to ensure the quality and effectiveness of machine learning models:\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Evaluate model output\"})}),/*#__PURE__*/e(\"p\",{children:\"After training and testing the model, labelers review and analyze the model outputs in conjunction with the ground truth labels. They calculate and interpret the evaluation metrics to assess model performance accurately.\"}),/*#__PURE__*/e(\"p\",{children:\"Experts compare and make informed decisions about which model performs best based on the evaluation metrics. Commonly used metrics encompass:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"accuracy - the ratio of correct predictions;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"precision - true positives over total predicted positives;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"recall - true positives divided by total predicted positives;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"F1 score - harmonic mean of precision and recall;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"area under the receiver operating characteristic curve (AUC-ROC).\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Data scientists consider these or some other task-specific metrics to select the model that aligns with the project's goals. Also, humans bring domain-specific knowledge to the table, which is often crucial for interpreting model outputs. They can identify cases where the model may be making false positives or negatives due to domain-specific nuances. In situations where models struggle to make accurate predictions, experts can provide additional guidance and insight. They can review and potentially re-label ambiguous or challenging data points.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Fine-tune open-source model\"})}),/*#__PURE__*/e(\"p\",{children:\"Open-source models are often trained on generic sets and may not be tailored to the specific requirements of your task. Human labeler can curate a specific set that precisely reflects the nuances, context, and goals of your project, ensuring that the model's performance is aligned with your specific needs.\"}),/*#__PURE__*/e(\"p\",{children:\"Such training sets are called gold standard datasets. They consist of high-quality, accurately labeled data that serves as a reference or benchmark for model training and evaluation. Fine-tuning a model on such sets allows an automated labeling system to adapt to the specific characteristics and requirements of a particular task or dataset.\"}),/*#__PURE__*/e(\"p\",{children:\"Humans can provide accurate and high-quality annotations for the gold standard set. These annotations serve as a reference or benchmark for training and evaluating the model. High-quality data is essential for achieving the desired level of model performance.\"}),/*#__PURE__*/e(\"p\",{children:\"Fine-tuning allows the incorporation of domain-specific knowledge and expertise into the automated labeling process. Experts can guide the fine-tuning process by providing feedback on model predictions, addressing domain-specific challenges, and mitigating potential biases.\"}),/*#__PURE__*/e(\"p\",{children:\"Creating a golden set also allows for iterative refinement. As you fine-tune the model and gain insights from its performance, you can revisit, update, or modify the golden set to improve system accuracy over time. This approach is invaluable for achieving successful results in machine learning projects with the use of open-source models.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Model-Assisted Labeling with the Help of LLM\"})}),/*#__PURE__*/e(\"p\",{children:\"LLM is a valuable asset for text-specific tasks and data labeling as well. LLMs can generate annotations quickly, significantly reducing the time required for labeling compared to manual annotation. However, human oversight remains essential for quality control and ensuring the accuracy of annotations, especially when it comes to complex tasks where models may not perform adequately on their own.\"}),/*#__PURE__*/e(\"p\",{children:\"Complex tasks often involve ambiguous content or borderline cases that LLMs may have issues with. Humans can resolve ambiguity and provide nuanced annotations based on their judgment and expertise. They play a critical role in QA by reviewing and verifying the accuracy of the labels generated by the LLM, ensuring that the data meets the high standards of the project.\"}),/*#__PURE__*/e(\"p\",{children:\"By working alongside an LLM, users can provide feedback to improve the model's performance. They can identify areas where the machine learning algorithms make errors and suggest corrective actions or refinements. It can also be highly cost-effective, as it reduces the need for extensive manual labeling and model training iterations.\"}),/*#__PURE__*/e(\"h2\",{children:\"Acquiring Labeled Data with Large Language Models\"}),/*#__PURE__*/e(\"p\",{children:\"Labeling with the help of large language models involves using them to generate annotations or labels for various types of content, primarily textual. It mainly works the following way:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Users provide an input prompt or instruction to the LLM, specifying the task and what they want the LLM to annotate;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"LLM generates initial annotations or labels for the presented data, which helps expedite the labeling process.  \"})})]}),/*#__PURE__*/e(\"p\",{children:\"At first glance, all looks great: data scientists obtain a labeled dataset for further processing as a result of LLM annotation. However, that's not all. It's essential to have annotators review and validate the labels generated by the LLM. Reviews ensure that the generated annotations are accurate and meet the desired quality standards.\"}),/*#__PURE__*/e(\"p\",{children:\"The reason for this is that while large language models are powerful and versatile, they are not without limitations and challenges that can make them imperfect and often unsuitable for practical applications in certain contexts. Here are some of the key reasons why large language models may have limitations:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"LLMs can inherit biases present in their training sets. In specialized applications, these biases may lead to corrupted or inappropriate responses, affecting correctness and fairness;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"LLMs are primarily trained on generic texts, which may not include or include not enough domain-specific or industry-specific knowledge. In specialized tasks, such as medical diagnosis or legal document analysis, this lack of domain expertise can result in incorrect or insufficient information. \"})})]}),/*#__PURE__*/e(\"p\",{children:\"Specialized tasks often require a substantial amount of annotated data to train accurate models. For instance, specific or business-oriented ones may include industry slang or terminology that LLMs may not understand or use accurately. The simple fact that they may lack access to a sufficient amount of such information, makes them less effective for these tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"To address these limitations, businesses often require a combination of LLMs and expert input to ensure correctness, reliability, and practicality in business-specific tasks. In order for LLMs to thoroughly perform tasks for particular companies they need to be fine-tuned on additionally labeled domain-specific datasets that are tailored to these organizations\u2019 needs.\"}),/*#__PURE__*/e(\"p\",{children:\"Additional annotation can significantly enhance the overall performance of LLMs by addressing some of their limitations and improving their capabilities, making them more effective in domains or applications where they may not perform optimally out of the box. Customized LLMs are more practical and effective in delivering domain-specific content and responses.\"}),/*#__PURE__*/e(\"p\",{children:\"Labels provided by domain experts help LLMs understand and generate more accurate and relevant content, thereby enhancing performance and correctness. By continually updating and refining annotations based on real-world usage and feedback, LLMs can undergo iterative improvement, leading to better performance and practicality over time.\"}),/*#__PURE__*/e(\"h2\",{children:\"The Future of AI-Assisted Data Labeling\"}),/*#__PURE__*/e(\"p\",{children:\"AI-assisted annotation is already helping to streamline manual labeling, however, it is not yet possible to completely eliminate the help of trained professionals. This is due to the fact that the current AI-assisted labeling methods, though having evolved and improved their capabilities over the last few years, are still not able to handle all challenges properly. This is especially true for applications involving complex issues that are specific to certain industries, including business.\"}),/*#__PURE__*/e(\"p\",{children:\"The current state of technology cannot eliminate the need for human involvement in data labeling. As AI models continue to improve, we can expect even more sophisticated AI-assisted labeling tools to emerge, further revolutionizing the field of AI and enabling groundbreaking applications across various domains.\"}),/*#__PURE__*/e(\"p\",{children:\"While it's challenging to predict the future of technology with certainty, it's probable that we may see more automation and reduced labelers' involvement in annotation. The extent to which complete automation will be achieved will likely vary depending on the multitude of considerations. Nonetheless, even as technology progresses, human experience, judgment, and supervision will likely continue to be required to ensure accuracy and quality in many data labeling scenarios.\"})]});export const richText4=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Fine-tuning is regarded as one of the optimal options for improving the performance of AI foundation models. However, there are other more cost-effective and efficient means of customizing large language models (LLMs), such as prompt tuning or prompt engineering. In this article, we will delve into the concept of such techniques and how they can empower data scientists to tackle the full potential of AI models for various applications.\"}),/*#__PURE__*/e(n,{href:\"https://toloka.ai/large-language-models/?utm_source=blog&utm_medium=banner&utm_campaign=sft&utm_content=llmsoptimizationtechniques\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{className:\"framer-image\",\"data-preset-tag\":\"img\",children:/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"240\",src:\"https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png\",srcSet:\"https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png?scale-down-to=512 512w,https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png 1498w\",style:{aspectRatio:\"1498 / 480\"},width:\"749\"})})}),/*#__PURE__*/e(\"h2\",{children:\"Understanding Options for Tailoring a Pre-Trained Model for a Designated Task\"}),/*#__PURE__*/e(\"p\",{children:\"Prompt tuning and engineering are the processes that focus on crafting a specific input or instruction (a prompt) for an AI model to obtain desired outputs. These approaches are essential because, despite their remarkable capabilities, AI models like LLMs can be unpredictable and may not always generate the pertinent response without precise guidance. So, basically, these techniques are employed to improve the performance of pre-trained LLMs to adapt it to downstream tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"Let's take a closer look at all the basic optimization techniques that exist to customize the model for a particular task.\"}),/*#__PURE__*/e(\"h2\",{children:\"Fine-Tuning\"}),/*#__PURE__*/e(\"p\",{children:\"One of the most prominent and effective methods to improve the capabilities of LLMs is a fine-tuning process. During fine-tuning, models receive additional training data that consists of labeled examples of the desired future output of the large language model.\"}),/*#__PURE__*/e(\"p\",{children:\"However, while fine-tuning most often requires a significant amount of new data, the process is much easier than training a new large language model. Other easier-to-perform alternatives of this approach include prompt tuning and prompt engineering.\"}),/*#__PURE__*/e(\"h2\",{children:\"Prompt Tuning and Engineering\"}),/*#__PURE__*/e(\"p\",{children:\"Prompt tuning and engineering allow enhancing LLM performance to handle highly specialized tasks without collecting a huge amount of labeled data as in the case of fine-tuning. These techniques may prove to be particularly valuable tools, for example, for businesses that possess limited data collection. Let's take a closer look at both of these approaches.\"}),/*#__PURE__*/e(\"h3\",{children:\"Prompt Engineering\"}),/*#__PURE__*/e(\"p\",{children:\"Prompt engineering is a process that allows to engineer guidelines for a pre-trained model to implement a narrow task. A human engineer's instructions are fed to an LLM for it to accomplish a specific task. These instructions are called hard prompts.\"}),/*#__PURE__*/e(\"p\",{children:\"Hard prompts are described as static and well-defined guidelines for an LLM. They can be thought of as templates that are used in generative AI applications. These hard prompts consist of manually handcrafted text prompts with discrete input tokens. They may represent some extra words, instructions, or examples for a model written by a human. They compel an LLM to retrieve an appropriate and relevant output from its already huge amount of knowledge.\"}),/*#__PURE__*/e(\"p\",{children:\"Some tasks require just one or two of such instructions for the model to successfully perform the task. But there are also more demanding tasks, which may need dozens of such hard prompts. It's hard to write all of them manually, so that's why a more effective and fast approach known as prompt tuning appeared. AI-crafted soft prompts used in prompt tuning tend to outperform human-engineered hard prompts.\"}),/*#__PURE__*/e(\"h3\",{children:\"Prompt Tuning\"}),/*#__PURE__*/e(\"p\",{children:\"LLMs with billions of parameters can be quite demanding to fine-tune for specific tasks due to their size and complexity. Traditional methods of task-specific fine-tuning might require significant computational resources and labeled data. Prompt tuning provides a more efficient way to adapt these LLMs to various tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"The so-called soft prompts are fed to LLM in the process of prompt tuning so that the model would understand the task-specific context. More commonly they represent an AI-generated number that is added to an embedding layer of deep learning models. Unlike hard prompts, soft prompts cannot be recognized or edited by a human, because they consist of an embedding that is a string of numbers.\"}),/*#__PURE__*/e(\"p\",{children:\"Such lists of numbers are generated by a small trainable model before the LMM is involved in the process. They are created through the text prompt encoding process. Then this small model creates task-specific virtual tokens that correspond to the embedding consisting of a string of numbers. The tokens are then added to the prompt that is fed to the model.\"}),/*#__PURE__*/t(\"p\",{children:[\"According to \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/abs/2104.08691?source=post_page-----49740de6c64c--------------------------------\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"research\"})}),\", as model parameters scale, prompt tuning tends to keep up with the performance of a traditional method of tuning a model. The main disadvantage of soft prompts compared to hard prompt parameters consists in the fact that they are less likely to be interpreted. This lack of interpretability does not allow us to fully understand how or why the AI language model chooses the specific soft prompt for a particular task.\"]}),/*#__PURE__*/e(\"p\",{children:\"Moreover, the model itself is not capable of sharing insights into its inner process of choosing these task-specific virtual tokens. Simply put, AI-designed soft prompts are not transparent just the same as the intricate layers of deep learning models themselves.\"}),/*#__PURE__*/e(\"p\",{children:'There is also a technique that is similar to prompt tuning and engineering which is referred to as prefix tuning. It also represents a streamlined substitute for full tuning in the context of natural language generation tasks. This approach maintains the existing language model weights frozen without further adjustments but focuses on optimizing a compact, specialized vector referred to as the \"prefix.\"'}),/*#__PURE__*/e(\"p\",{children:\"Instead of just changing words or instructions in the input, prefix tuning adds some extra information right at the beginning of the prompt. This extra information helps the language model understand your request better.\"}),/*#__PURE__*/e(\"h2\",{children:\"Why Optimization Techniques Matter\"}),/*#__PURE__*/e(\"p\",{children:\"Introducing hard or soft prompts to an LLM may be a less expensive and more effective approach to induce a pre-trained model to undertake a specific task than fine-tuning. For example, a good language classifier prompt can substitute a vast amount of additional training data and guide the model to the accurate desired output. Certain advantages make prompt optimization techniques stand out in comparison to traditional methods of improving the capabilities of LLMs:\"}),/*#__PURE__*/e(\"h3\",{children:\"Customization\"}),/*#__PURE__*/e(\"p\",{children:\"It's possible to craft prompts to suit various tasks, making them versatile for a wide range of applications, from question answering to content generation and more. Optimization techniques enable you to personalize your interactions with AI models, ensuring that responses are tailored to your goals and objectives.\"}),/*#__PURE__*/e(\"h3\",{children:\"Efficiency\"}),/*#__PURE__*/e(\"p\",{children:\"When prompts are accurately tuned or engineered, AI performance becomes more efficient. The need for post-processing or manual editing is minimized, saving valuable time and resources. Both prompt tuning and engineering provide efficient ways to customize LLMs. They allow users to adapt existing models to specific tasks without the need for thorough large-scale retraining or fine-tuning. This efficiency saves time and computational resources.\"}),/*#__PURE__*/e(\"h3\",{children:\"Parameter Efficiency\"}),/*#__PURE__*/e(\"p\",{children:\"These methods are parameter-efficient. They don't require the creation of a whole new large base model or modification of the entire model but focus on specific components, such as prompts or prefixes. They also don't require any additional datasets. This means they can work effectively even with limited computational resources and restricted labeled data.\"}),/*#__PURE__*/e(\"h3\",{children:\"Reduced Data Dependency\"}),/*#__PURE__*/e(\"p\",{children:\"Prompt tuning and engineering can reduce the data dependency of pre-trained language models. They allow models to perform well with relatively small amounts of data, which can be especially important for specialized tasks with limited training examples.\"}),/*#__PURE__*/e(\"h3\",{children:\"Reusable Models\"}),/*#__PURE__*/e(\"p\",{children:\"Prompt tuning and engineering help to find new applications for trained models and solve a variety of tasks. This reduces the need to maintain separate models for each specific application, thus saving on data storage and computational costs.\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/e(\"p\",{children:\"Prompt tuning and engineering are artificial intelligence model optimization techniques that facilitate efficient and flexible refinement of language models. They make LLMs more user-friendly and adaptable to a wide variety of different specific tasks. In a nutshell, it\u2019s a simple yet powerful way to make large language models more intelligent and multifunctional. In certain cases, when, for example, the amount of labeled data or computational resources is limited, they may prove to work better than older additional training techniques.\"}),/*#__PURE__*/e(\"p\",{children:\"Prompt tuning can optimize LLMs through the introduction of AI-generated soft prompts, while prompt engineering, on the other hand, provides a sense of control, enabling users to craft precise hard prompts for desired outcomes. Although soft prompts are prone to outperform human-generated hard prompts, the implementation of hard prompts is sufficient for the successful completion of certain trivial tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"As AI technology continues to evolve, prompt tuning and engineering will likely remain critical skills for those looking to leverage AI for various applications, from content generation to problem-solving and beyond.\"})]});export const richText5=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"In the ever-evolving landscape of software development, developers are constantly seeking ways to increase efficiency and productivity. Generative artificial intelligence has the potential to revolutionize the way they write code, debug, and solve coding challenges. Large Language Models (LLMs) are prime examples of generative AI models, as they can generate coherent and contextually relevant text including code, based on the natural language prompts they receive.\"}),/*#__PURE__*/t(\"p\",{children:[\"Toloka's support to \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/blog/bigcode-project/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"BigCode\"})}),\" project showcases the tremendous potential of LLMs and crowd collaboration in the sphere of coding. We will explore how LLMs are transforming coding and highlight some other noteworthy applications.\"]}),/*#__PURE__*/e(\"h2\",{children:\"What are code-generating LLMs\"}),/*#__PURE__*/e(\"p\",{children:\"A large language model is a deep-learning model that knows how to generate relevant textual information and thus represents a vivid example of generative AI. Deep means that the neural network that is the base of LLM has a lot of layers, which gives such a system an extensive array of capabilities. For example, it can memorize a huge amount of facts and insights about the world, predict a large number of words following each other in a text, and so forth.\"}),/*#__PURE__*/e(\"p\",{children:\"Code-generating LLMs are specialized AI models designed for the task of generating computer code. Such models undergo training with datasets that encompass repositories, specialized documentation, programming code, relevant web content, and other related data. They possess a deep understanding of various programming languages, coding patterns, and software development concepts. Code LLMs are engineered to assist developers and programmers.\"}),/*#__PURE__*/e(\"p\",{children:\"When these LLMs are seamlessly integrated into the developer's Integrated Development Environment (IDE), they possess a comprehensive understanding of the coding context, which includes elements such as comments, function names, and variable names. This contextual awareness empowers them to provide more refined and contextually relevant coding suggestions.\"}),/*#__PURE__*/e(\"h2\",{children:\"Capabilities of LLMs in Coding\"}),/*#__PURE__*/e(\"p\",{children:\"Initially, LLMs were intended to generate various textual data. They accomplish the tasks of composing, translating texts, and can respond to queries posed by a human and others. LLMs have demonstrated remarkable capabilities in understanding and generating human-like text. And now they can also make code suggestions, as it is also a language, but designed specifically for programming.\"}),/*#__PURE__*/e(\"p\",{children:\"These models are fine-tuned for specific tasks, and when it comes to coding, their capabilities are staggering. LLMs have the potential to significantly boost developer\u2019s productivity, reduce coding errors, and enhance the overall coding experience. Generative AI, LLMs in particular, offer numerous ways to assist developers across various aspects of their work:\"}),/*#__PURE__*/e(\"h3\",{children:\"Code Generation\"}),/*#__PURE__*/e(\"p\",{children:\"LLMs excel at generating code snippets or entire functions based on natural language descriptions. Developers can simply describe what they want to achieve in plain language, and the LLM can generate code that corresponds to their needs. This functionality significantly reduces the entry barrier for coding, making it more accessible to non-programmers.\"}),/*#__PURE__*/e(\"h3\",{children:\"Code Completion\"}),/*#__PURE__*/e(\"p\",{children:\"Code completion, often referred to as autocompletion, is one of the AI coding tools that automatically suggest code snippets and other elements as developers type. Autocompletion reduces the need to write lengthy code constructs manually. By offering contextually relevant suggestions, it speeds up the coding process. Developers can write code quickly and more efficiently. This not only saves time but also minimizes the risk of typos and syntax errors.\"}),/*#__PURE__*/e(\"h3\",{children:\"Code Translation and Modernization\"}),/*#__PURE__*/e(\"p\",{children:\"LLMs are trained on extensive multilingual text data, making them proficient in multiple programming languages. They can easily understand and translate existing code from one language to another, aiding in interoperability between different systems and languages, because different programming languages may not seamlessly integrate with one another. Code translation is necessary to ensure that components written in different languages can work together effectively. Also, older projects may be written in outdated or less commonly used languages. Code translation can modernize such code.\"}),/*#__PURE__*/e(\"h3\",{children:\"Code Summarization\"}),/*#__PURE__*/e(\"p\",{children:\"Software projects involve extensive codebases, libraries, and frameworks. LLMs are good at understanding natural language, including code comments and documentation. They can analyze and explain code, providing concise summaries. Code summarization makes codebases more readable by extracting key insights, function descriptions, and usage patterns, reducing the load on developers. Maintaining and updating code is more efficient when developers can quickly comprehend its purpose and functionality. Broadly speaking, \u0441ode summarization can accelerate developer productivity\"}),/*#__PURE__*/e(\"h3\",{children:\"Code Refactoring\"}),/*#__PURE__*/e(\"p\",{children:\"Refactoring code makes it easier to read. Maintaining clean and efficient code is crucial for software projects. LLMs can offer suggestions for code refactoring, helping developers improve code performance. Through refactoring, developers can identify and eliminate inefficiencies.. This optimization can result in faster execution and reduced resources consumption.\"}),/*#__PURE__*/e(\"h3\",{children:\"AI-Assisted Testing\"}),/*#__PURE__*/e(\"p\",{children:\"Quality assurance and testing are integral parts of the software development lifecycle. Generative AI can assist in creating test cases, ensuring code correctness and robustness. LLMs can automatically generate such tests based on code descriptions or specifications, saving time and effort in test case creation. By automating test case generation, enhancing test data creation, and proactively identifying issues and vulnerabilities, AI-driven testing not only ensures software quality but also accelerates development cycles\"}),/*#__PURE__*/e(\"h3\",{children:\"Error Detection and Debugging\"}),/*#__PURE__*/e(\"p\",{children:\"LLMs can analyze existing code and recognize errors, including errors in syntax or variables and logical inconsistencies, saving developers valuable time in the error identification phase. Generative AI can provide detailed descriptions of errors, explaining their causes and suggesting potential fixes, making it easier for software developers to understand and address issues.\"}),/*#__PURE__*/e(\"p\",{children:\"With LLM-powered code suggestions, developers can code faster, more accurately, and with a deeper understanding of their projects, ultimately accelerating software development. While LLMs offer powerful assistance, developers should also consider the quality of the generated code, potential ethical concerns, and data privacy and security when using generative AI.\"}),/*#__PURE__*/e(\"h2\",{children:\"Advantages of LLMs for Coding\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Efficiency and speed\"}),\". LLMs significantly boost coding efficiency by providing coding suggestions and autocompletions. Generative AI can also greatly accelerate the coding process;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Reduced errors\"}),\". By automating routine tasks and offering real-time error prevention, LLMs help reduce human errors, leading to fewer bugs and improved software quality;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Developers productivity\"}),\". Developers can focus on higher-level problem-solving as LLMs handle repetitive tasks for coding. This results in increased overall productivity;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Accessibility and education\"}),\". LLMs with natural language interfaces make coding more accessible to non-programmers, allowing them to describe it in plain language. That way LLMs can serve as educational tools, helping beginners learn programming concepts and best practices by interacting with AI models;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Versatility of use\"}),\". LLMs are versatile and support multiple languages and programming tasks. This means developers can use the same tool for various coding needs, further speeding up the development process.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"With LLM-powered code suggestions, developers can work faster, more accurately, and with a deeper understanding of their projects, ultimately accelerating software development. While generative AI offers powerful assistance, developers should also consider the quality of the generated code, potential ethical concerns, and data privacy and security when using generative AI tools.\"}),/*#__PURE__*/e(\"h2\",{children:\"Possible Challenges of Code Generation\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI coding tools can significantly enhance productivity, code quality, and understanding, but they also come with certain challenges and considerations:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Code Quality\"}),\". The LLM code quality may vary. Therefore, developers must review and validate it to ensure that it meets their standards;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Data Privacy and Security\"}),\". Using LLMs for code creation and analysis may involve sharing sensitive data. LLMs may inadvertently introduce security vulnerabilities if not used with care. Ensuring the security of the source code is imperative, so developers should take precautions to \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/blog/bigcode-project/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"protect data\"})}),\" privacy and security;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Model Biases\"}),\". LLMs may exhibit biases present in their training data, potentially leading to biased code suggestions or documentation. Careful review and bias mitigation strategies are necessary;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Over-Reliance\"}),\". There's a risk of over-reliance on LLMs, which could lead to developers depending too heavily on AI code without fully understanding the logic behind it.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"LLMs offer significant advantages in terms of coding efficiency, quality, and productivity. However, they come with potential drawbacks related to code quality, security, bias, and the risk of overreliance. Developers and organizations should use LLMs judiciously, exercise proper oversight, and ensure that new code aligns with project requirements and coding standards\"}),/*#__PURE__*/e(\"p\",{children:\"There are occasions when generation by means of LLMs cannot be accomplished without human help. Even though generative AI generally can handle most tasks, in some specific coding cases it requires human assistance.\"}),/*#__PURE__*/e(\"h2\",{children:\"Tasks Requiring Developer Expertise\"}),/*#__PURE__*/t(\"p\",{children:[\"Surely, generative AI shows great promise in coding, yet a research held by \",/*#__PURE__*/e(n,{href:\"https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/unleashing-developer-productivity-with-generative-ai\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"McKinsey\"})}),\" highlights the importance of skilled engineers in utilizing these tools effectively. While accomplishing the tasks of code development using generative AI tools for the study, the test group determined three critical areas where human oversight and expertise are indispensable:\"]}),/*#__PURE__*/t(\"ol\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\",\"--list-style-type\":\"unset\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Code Quality Assurance\"}),\". Participants reported instances where generative AI-based tools provided coding recommendations that were incorrect or introduced errors into the code. For example, one developer had to input multiple prompts to correct an AI tool's mistaken assumptions, while another had to meticulously guide the tool through debugging processes. Thus, we can conclude that experts play a vital role in thoroughly examining code for bugs and errors;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Contextual Understanding\"}),\". Effective work is impossible if an employee is not familiar with the company they work for. We can also say the same about generative AI coding tools. They lack awareness of a specific project's and organization's needs in their \\\"as-is\\\" state. Developers must provide these tools with contextual information, including how the code will be utilized, the interfaces and systems it will interact with, the data employed, and more, as emphasized by research participants. Understanding the context of a coding task is essential to ensure that the final software product aligns with other company's applications, meets performance and security requirements, and addresses end-user needs;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"**Complex Problem Solving. Generative AI-based tools excel at responding to straightforward natural language prompts, such as code optimization tasks. However, they may struggle with intricate requirements that involve combining multiple frameworks with diverse code logic. To address multifaceted demands, developers often need to manually merge components or break the code into smaller pieces. As one participant pointed out, generative AI is less effective when tackling complex problems that require a holistic approach.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"So, there's no doubt that generative AI holds tremendous potential to support developers in various coding tasks, but still, there are some issues that cannot be fully solved by coding LLMs alone. The collaboration between AI technology and human expertise here is pivotal in order to experience the full potential of generative AI tools in software development.\"}),/*#__PURE__*/e(\"p\",{children:\"In addition to the issues outlined, modern LLMs for coding are affected by a lack of personal data protection. To prevent any kind of personal data leaks from occurring in the future, the BigCode project came up with the idea of creating a model for identifying and removing personal data from the training datasets for such LLMs.\"}),/*#__PURE__*/e(\"p\",{children:\"To label data for such a model, tolokers have taken part in the project, since no one but humans can better label the necessary information considered personal data so efficiently and quickly. Human annotators can provide the contextual understanding, quality assurance, and domain-specific knowledge required for accurate and reliable annotation, particularly when dealing with sensitive data in code snippets.\"}),/*#__PURE__*/e(\"h2\",{children:\"Toloka's BigCode Project: LLM needs human assistance\"}),/*#__PURE__*/e(\"p\",{children:\"The BigCode project is a collaborative effort led by HuggingFace and ServiceNow aimed at enhancing the coding capabilities of Large Language Models (LLMs) by addressing regulatory challenges and improving data privacy. The project received assistance from Toloka to address specific challenges related to the handling of sensitive information within their dataset which contains a vast amount of code written in multiple programming languages, totaling 6.4 terabytes.\"}),/*#__PURE__*/e(\"p\",{children:\"One of the primary objectives of the BigCode project was to ensure compliance with data privacy regulations, particularly GDPR (General Data Protection Regulation). This involved the removal of sensitive personal information, such as usernames, passwords, and security tokens, from the dataset to safeguard user privacy and data security.\"}),/*#__PURE__*/e(\"p\",{children:\"To achieve this goal, the project curators needed to create a personal identifiable information (PII) model training dataset that could automatically detect and mask personal data within the dataset They identified 14 distinct categories of sensitive data that needed to be accurately recognized and handled appropriately.\"}),/*#__PURE__*/e(\"p\",{children:\"However, annotating and labeling such a massive dataset would be a huge challenge. Traditionally, this task would have required hiring a team of skilled programmers who could manually review and label each code snippet according to the identified categories. Given the size of the dataset, consisting of 12,000 code chunks, this would have been a time-consuming and resource-intensive process, potentially taking many months.\"}),/*#__PURE__*/e(\"p\",{children:\"Toloka introduced an innovative and non-traditional approach to tackle the challenge. They used the power of crowdsourcing by engaging a diverse group of 1399 tolokers from 35 different countries. This effort just took four days and amounted to a total of 4349 person-hours. To put this into perspective, it was equivalent to the workload of a single programmer working diligently for an entire year.\"}),/*#__PURE__*/e(\"p\",{children:\"In the context of the BigCode project, it becomes evident that while LLMs can be valuable generative AI tools for assisting in the creation of code, human supervision and expertise remain essential. This interesting case also illustrates that LLMs though already very powerful and versatile in terms of tasks they can accomplish, still have weaknesses. They do need human refinement and verification of the information they generate. Still, even now generative AI tools significantly speed up the work of developers if they approach the task properly.\"}),/*#__PURE__*/e(\"h2\",{children:\"Generative AI coding tools\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI coding tools are software applications and platforms that utilize the power of generative AI, typically using Large Language Models, to assist software developers in various coding-related tasks. These tools aim to streamline the software development process, increase productivity, and enhance code quality. Here's a list of some notable generative AI tools:\"}),/*#__PURE__*/e(\"h3\",{children:\"GitHub Copilot\"}),/*#__PURE__*/e(\"p\",{children:\"Developed by GitHub in collaboration with OpenAI, GitHub Copilot is an AI-powered coding assistant that integrates directly into popular editors like Visual Studio Code. It provides real-time autocompletions and even generates entire code examples based on natural language comments and descriptions. It can also help with boilerplate code for unit tests generation.\"}),/*#__PURE__*/e(\"h3\",{children:\"AlphaCode\"}),/*#__PURE__*/e(\"p\",{children:\"The AlphaCode neural network created by DeepMind can solve tasks at the software engineer level with a few months of experience. AlphaCode utilizes transformer-based language models for code generation. DeepMind experts pointed out that their product turned out to have the ability to independently write program code without having structural information about algorithms or program language.\"}),/*#__PURE__*/e(\"h3\",{children:\"TabNine\"}),/*#__PURE__*/e(\"p\",{children:\"TabNine is an AI-powered autocompletion tool that uses GPT-based models to predict and suggest code as developers write. The tool analyzes the context in which the code is used and generates the most appropriate fragments. It supports a wide range of programming languages and Integrated Development Environments (IDEs), making it a versatile choice for developers.\"}),/*#__PURE__*/e(\"h3\",{children:\"Amazon CodeWhisperer\"}),/*#__PURE__*/e(\"p\",{children:\"The CodeWhisperer is a programming tool developed by Amazon, that can automatically write entire functions based on just a comment or a few lines of code. Developers can use it directly in their preferred IDEs, including IntelliJ IDEA, PyCharm, WebStorm, and Amazon's proprietary AWS 9 cloud. The company points out that the system constantly reviews your code and comments, and even takes into account your own programming style and variable names.\"}),/*#__PURE__*/e(\"h3\",{children:\"ChatGPT\"}),/*#__PURE__*/e(\"p\",{children:\"While GPT (Generative Pre-trained Transformer) models are not specifically designed as coding tools, they are a class of powerful generative AI models that can be adapted for various tasks, including coding-related tasks. ChatGPT can assist in generating code snippets based on the input and context provided. The quality and accuracy of the result generated by ChatGPT depend on the clarity of the input instructions and the complexity of the coding task.\"}),/*#__PURE__*/e(\"h3\",{children:\"Google Bard\"}),/*#__PURE__*/e(\"p\",{children:\"Google has updated its Bard AI chatbot to help developers write and debug code. Bard is capable of generating new code, debugging, helping explain lines of coding, and even writing functions for Google Sheets in more than 20 languages, including C++, Go, Java, Javascript, Python, and Typescript\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI for coding represents a significant advancement in software development, making coding more accessible, efficient, and collaborative. The advantages are evident: increased coding speed, the capability to improve coding quality and reduce errors, as well as an opportunity to enhance overall productivity. It empowers developers to focus on higher-level problem-solving, creativity, and innovation, while routine and repetitive coding tasks are automated.\"}),/*#__PURE__*/e(\"p\",{children:\"However, as with any powerful technology, there are considerations and challenges. Code quality, security, and potential biases must be carefully managed. Developers should exercise due diligence and human oversight to ensure that the generated coding suggestions align with project requirements and best practices.\"}),/*#__PURE__*/e(\"p\",{children:\"Human assistance in the creation and development of code LLMs is a critical factor. These models are not standalone entities but are the result of extensive human involvement. The training process involves data supervision, labeling, and continuous fine-tuning by experts.\"}),/*#__PURE__*/e(\"p\",{children:\"Prominent examples of generative AI coding tools include, for example, GitHub Copilot and TabNine, both of which are designed to assist developers in writing code more efficiently and with higher quality. These models have the potential to significantly streamline the software development process, making it faster, more accessible, and less error-prone. As AI technologies continue to evolve, we can expect even more innovative applications and tools to further enhance the developer experience.\"})]});export const richText6=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"In the fast-evolving landscape of AI, Large Language Models (LLMs) stand out as the most rapidly progressing subfield, offering transformative potential for business applications. Just over the past month, we've seen a flurry of activity in this space. New tech startups are sprouting up, and established players are unveiling their LLM offerings. For instance, EY recently \",/*#__PURE__*/e(n,{href:\"https://www.ey.com/en_gl/news/2023/09/ey-announces-launch-of-artificial-intelligence-platform-ey-ai-following-us-1-4b-investment\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"launched\"})}),\" its EY.AI platform, marking their confidence in this tech with a staggering $1.4 billion investment. While the market heats up, it's still up for debate whether supply is outpacing demand. \"]}),/*#__PURE__*/e(\"p\",{children:\"From my point of view, there remains a strong case for developing a custom LLM tailored to the distinct needs of your business. The case is even stronger if your company has a considerable amount of data that you can leverage, or if your data science team has tested generic solutions for your business cases and you see an open opportunity in the market.\"}),/*#__PURE__*/e(\"p\",{children:\"Recently, we at Toloka AI embarked on an exciting journey: crafting a new LLM from the ground up. We are collaborating with another AI startup as a data partner responsible for data collection, refinement, evaluation and continuous monitoring. Through this process, we've gained invaluable insights to share with you. And trust me, there's more on the horizon.\"}),/*#__PURE__*/e(\"p\",{children:\"This post kicks off a series delving into the nuts and bolts of building an LLM from scratch, but today we want to present the big picture: where do you begin? Before a single line of code is written or a single data point is collected, it's imperative to outline your vision for the model. Let's dive into the first step: crafting the tech specification for your LLM.\"}),/*#__PURE__*/e(\"h2\",{children:\"Project scope and objectives\"}),/*#__PURE__*/e(\"p\",{children:\"It's crucial to identify the specific challenges your model is designed to address. Currently, expertise in particular languages is a notable differentiation. But suppose you're tailoring your LLM for specialized tasks, such as legal document processing or managerial decision support. In that case, this will influence your data sourcing and labeling strategies. It\u2019s wise to define your primary market niche early on: What niche will your LLM serve? What areas will it excel in? What unique value proposition are you aiming to offer?\"}),/*#__PURE__*/e(\"p\",{children:\"While the adaptability of LLMs is commendable, it can also be a double-edged sword. The market is already saturated with general-purpose LLMs. Thus, carving out specific business scenarios where your model can outperform competitors becomes an essential first move.\"}),/*#__PURE__*/e(\"h2\",{children:\"Data acquisition and management\"}),/*#__PURE__*/e(\"p\",{children:\"Data serves as the foundation of any LLM. It's vital to ensure that your data sources align with your objectives and come from a mix of diverse origins. Beyond mere collection, data often demands rigorous cleaning and preprocessing to meet quality standards. In today's digital landscape, prioritizing data privacy and ethical standards isn't just commendable\u2014it's non-negotiable.\"}),/*#__PURE__*/t(\"p\",{children:[\"Our \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/blog/bigcode-project/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"collaboration\"})}),\" with HuggingFace on the BigCode project underscored a fundamental truth: the quality of your data profoundly impacts your model's performance. If you possess proprietary data, list all available resources. Seek out open data that aligns with your earlier defined business scenarios. And if there's an opportunity to enhance your data with human insights, seize it. Your diligence will pay dividends in the final outcomes.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Safety and bias mitigation\"}),/*#__PURE__*/e(\"p\",{children:\"AI, for all its transformative potential, has its challenges. Biased outputs from LLMs can not only distort decisions but also cause serious damage and tarnish reputations. It's essential to implement robust measures to identify biases and introduce safeguards against undesired outputs, ensuring that the LLM remains compliant and trustworthy.\"}),/*#__PURE__*/t(\"p\",{children:[\"As we build our language model, we constantly face the challenge of detecting and mitigating various biases. If you want to dig deeper into this topic, see the recent \",/*#__PURE__*/e(n,{href:\"https://youtu.be/xIsmxYOMD3o?si=73PWxi5ix4vIiNMT\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"webinar\"})}),\" by Toloka developer advocate Prof. Ujwal Gadiraju. Cultural nuances heavily influence design choices. Here, human perspective is indispensable. In upcoming posts, we'll explore how diverse \\\"Tolokers\\\" identify distinct biases that must be addressed before deployment. While enlisting ethicists provides valuable insights, it's no silver bullet. A blended approach of crowdsourcing that taps into varied cultural and educational backgrounds, paired with expert guidance, offers the most holistic assurance of quality.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Evaluation metrics and benchmarks\"}),/*#__PURE__*/e(\"p\",{children:\"To effectively assess your LLM's performance, it's essential to have well-defined metrics and benchmarks. Whatever evaluation methodology you prefer, be it evaluation on some open or closed benchmark, some data labelling pipeline or evaluation with a stand-alone model, choose one and stick to it. This decision gives you clarity on your LLM capabilities. Additionally, comparing your LLM with existing solutions provides insights into its market standing.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://toloka.ai/blog/offline-evaluation/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Evaluation\"})}),\" remains a focal point in LLM research, with many initiatives underway to refine and bolster assessment methods. For those keen on this subject, stay tuned. We're on the cusp of presenting Toloka\u2019s innovative approach to LLM evaluation methodology. You also use evaluation to compare your LLM with existing solutions and gain insight into its market standing.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Monitoring and maintenance\"}),/*#__PURE__*/e(\"p\",{children:\"Post-deployment, the journey with your LLM is just beginning. Constant performance monitoring reveals areas for refinement. Additionally, as data landscapes shift and business demands evolve, scheduled model updates and retraining become vital.\"}),/*#__PURE__*/t(\"p\",{children:[\"For context, a recent \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/abs/2307.09009\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"study\"})}),\" from Stanford and Berkeley observed temporal shifts in ChatGPT's performance. Although ChatGPT is proprietary, prompting speculation rather than concrete conclusions, it's reasonable to hypothesize that changes might be linked to human feedback loops. While user feedback is invaluable for refining models, without vigilant oversight, it can lead to unforeseen challenges. If you don\u2019t trust me, read this cautionary \",/*#__PURE__*/e(n,{href:\"https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"tale\"})}),\" on how unchecked feedback made a chatbot produce inappropriate content in under 24 hours.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion and future directions\"}),/*#__PURE__*/e(\"p\",{children:\"Embarking on the path to craft a custom LLM is both ambitious and rewarding, with the capacity to significantly elevate your business operations. But before delving into such a substantial investment, consider the following pivotal questions:\"}),/*#__PURE__*/e(\"p\",{children:\"\u2014 What's your niche? Where will your LLM fit within the market landscape? \u2014 How will you source and refine your data? Can you infuse human insights and secure datasets that lend your model a competitive edge? \u2014 What pitfalls must your model avoid? How can you ensure comprehensive coverage of potential undesired outcomes and mitigate risks? \u2014 Which evaluation methods will best measure your LLM's efficacy? \u2014 What strategies will underpin ongoing maintenance and optimize the feedback loop?\"}),/*#__PURE__*/e(\"p\",{children:\"By addressing these questions, you're setting the foundation for an LLM that's both innovative and grounded in strategic foresight.\"}),/*#__PURE__*/t(\"p\",{children:[\"Good luck! If you already have some answers, reach out to discuss \u2014 and consider Toloka as a \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/large-language-models/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"data partner\"})}),\" for your shiny new LLM.\"]})]});export const richText7=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"In the rapidly evolving landscape of AI technologies, data labeling plays a vital role in training machine learning models. Accurate and well-labeled data is the foundation of model performance. Traditionally, manual data labeling has been the go-to method, but it's progressively becoming outdated for modern enterprises.\"}),/*#__PURE__*/e(\"p\",{children:\"We'll further explore the evolution of data labeling, from manual to automated data labeling, and finally, the superior form of automated data labeling with Large Language Models (LLMs). We'll also delve into the concept of hybrid labeling, which combines human assistance with LLMs for the most desirable labeling result.\"}),/*#__PURE__*/e(\"h2\",{children:\"Manual Data Labeling: The Traditional Approach\"}),/*#__PURE__*/e(\"p\",{children:\"Manual labeling, also known as human annotation, is a fundamental procedure in data annotation and plays a crucial role in various machine learning projects and AI applications. It involves human labelers or annotators reviewing and assigning data labels or annotations to datasets based on specific criteria or guidelines.\"}),/*#__PURE__*/e(\"p\",{children:\"Manual labeling ensures the creation of high-quality labeled datasets, which are the foundation of powerful machine learning models. Annotators can apply domain expertise, contextual understanding, and common-sense reasoning to manifold labeling tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"While this method offers a high level of precision, it is labor-intensive, time-consuming, and expensive. Moreover, manual labeling is considered outdated in the context of modern data labeling and machine learning applications for modern enterprises.\"}),/*#__PURE__*/e(\"h3\",{children:\"Limitations of the manual labeling process\"}),/*#__PURE__*/e(\"p\",{children:\"One of the primary drawbacks of the manual labeling is its restricted scalability. With the exponential growth of data, many machine learning tasks require enormous datasets for training, testing, and validation. Manually labeling such vast amounts of data is lengthy and occasionally even impractical. Limited scalability of manual labeling makes it unsuitable for the ever-growing datasets required in AI applications today.\"}),/*#__PURE__*/e(\"p\",{children:\"Moreover, hiring and training human annotators, along with the time required for them to label data accurately, can result in significant expenses for organizations. This cost factor becomes a major deterrent, especially for startups and smaller enterprises with limited budgets.\"}),/*#__PURE__*/e(\"p\",{children:\"Manual labeling heavily depends on the availability of a skilled workforce. Organizations need to invest in recruiting, training, and managing annotators, which can be resource-intensive. However, labelers, no matter how experienced they are, are susceptible to inconsistencies. Even with guidelines and training, there can be discrepancies in how various labelers identify labels in the same data, leading to issues in the quality and reliability of labeled datasets.\"}),/*#__PURE__*/e(\"p\",{children:\"Manual data labeling is time-intensive. It can take days, weeks, or even months to annotate a large dataset, depending on its size and complexity. For repetitive labeling tasks, manual labeling is not only inefficient but also monotonous for annotators. This can lead to boredom and decreased accuracy over time.\"}),/*#__PURE__*/e(\"p\",{children:\"But still manual labeling is a critical component of data preparation in AI and machine learning projects. It excels in handling complex, nuanced, and context-dependent tasks, providing high-quality labeled datasets. However, it usually takes a lot of time, can be resource-intensive, and is subject to limited scalability and label variations.\"}),/*#__PURE__*/e(\"h2\",{children:\"Automated Data Labeling: A Step Towards Efficiency\"}),/*#__PURE__*/e(\"p\",{children:\"As organizations sought to overcome the limitations of the manual labeling process, they turned to automated data labeling solutions. These solutions often leverage rule-based algorithms and predefined guidelines to label raw data automatically. Auto-labeling is a capability commonly integrated into data annotation tools, utilizing artificial intelligence to automatically enhance or label a dataset.\"}),/*#__PURE__*/e(\"p\",{children:\"With the rise of machine learning algorithms, automating the assignment of labels to data with a high level of precision has become feasible. This process entails training a model on high-quality training datasets and then employing this model to label fresh, unlabeled data. Over time, the model refines its accuracy through exposure to more data, eventually achieving levels of precision comparable to manual labeling.\"}),/*#__PURE__*/e(\"p\",{children:\"In contrast to manual labeling, automated data labeling relies on machine learning algorithms to label data points efficiently and accurately. These algorithms can swiftly and precisely label extensive datasets, reducing the time and expense associated with manual labeling. Moreover, automated data labeling helps to mitigate the risk of human error and bias, yielding more uniform and dependable data annotations.\"}),/*#__PURE__*/e(\"h3\",{children:\"Advantages of Automated Labeling\"}),/*#__PURE__*/e(\"p\",{children:\"Automated labeling can significantly accelerate the labeling process, especially for large datasets, making it an essential tool for tasks related to natural language processing (NLP) or computer vision.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Speed and Efficiency.\"}),\" One of the primary advantages of automated labeling is its speed. Automated systems can label large volumes of data at a fraction of the time it would take human annotators. This efficiency is particularly valuable in applications that require quick data processing.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Scalability.\"}),\" Automated data labeling is highly scalable. It can handle substantial datasets without hiring and training a large team of annotators. This scalability is essential in machine learning applications that require extensive data for training.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cost Savings.\"}),\" Automated labeling can significantly reduce the expenses associated with data labeling. While developing and implementing automated systems may require certain investments, the long-term savings can be substantial, especially for organizations dealing with massive datasets.\"]}),/*#__PURE__*/e(\"p\",{children:\"Automated labeling can handle the processing of large datasets rapidly, making it highly scalable and cost-effective for projects with extensive data requirements. Nonetheless, automated data labeling comes with its own set of hurdles. For instance, the precision of automated data labeling is highly contingent on the qualitative characteristics of the training data and the intricacies of labeling data tasks. Furthermore, certain data types may pose challenges for automated labeling, such as images featuring ambiguous backgrounds or jokes in the text.\"}),/*#__PURE__*/e(\"p\",{children:\"Human oversight and manual review may still be necessary, especially for nuanced or domain-specific labeling tasks, to ensure the highest level of accuracy and reliability of labels. So, while automation brings efficiency and consistency, it can still struggle with complex and nuanced labeling objectives, often requiring a high degree of manual tuning to achieve acceptable accuracy.\"}),/*#__PURE__*/e(\"h2\",{children:\"Labeling with LLMs: The Superior Form of Automation\"}),/*#__PURE__*/e(\"p\",{children:\"Large Language Models are advanced AI models that have revolutionized data labeling. They use huge amounts of data and elaborate algorithms to understand, interpret, and create texts in human language. LLMs possess the ability to understand context, language nuances, and even the specific objectives of a labeling assignment. They are mostly built using deep learning techniques, most notably neural networks, which allow them to process and learn from substantial amounts of textual data.\"}),/*#__PURE__*/e(\"p\",{children:\"Utilizing LLMs to automate data labeling brings fantastic speed and stable quality to the process while simultaneously lowering labeling costs significantly. This is a kind of superior form of automated data labeling.\"}),/*#__PURE__*/e(\"p\",{children:\"Such advanced AI models, which are pre-trained on vast amounts of training data, have the capability to understand and generate human-like text, making them highly versatile tools for an extensive range of natural language processing tasks. LLMs convert raw data into labeled data by leveraging their NLP capabilities.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://toloka.ai/labeling-with-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Labeling data with LLMs\"})}),\" is possible at an incredible speed, far surpassing manual data labeling process and traditional automated systems. This accelerated pace is essential for organizations dealing with large and expanding datasets.\"]}),/*#__PURE__*/t(\"h3\",{children:[\"Key advantages of \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/labeling-with-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"automated data labeling with LLMs\"})})]}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Speed and Scalability. LLMs can label vast amounts of data in a fraction of the time it would take humans or traditional automated data labeling systems.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Cost-Efficiency. By automating labeling with LLMs, organizations can reduce labor costs and increase their labeling capacity without compromising on quality.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Adaptability. LLMs can handle a wide range of data labeling tasks, from simple classification to complex entity recognition, making them versatile tools for automated data labeling.\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Types of tasks performed by LLMs for automated data labeling purposes\"}),/*#__PURE__*/t(\"p\",{children:[\"LLMs can be utilized for \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/labeling-with-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"automated labeling\"})}),\" in various ways:\"]}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Text Classification. LLMs can classify text documents into predefined categories or assign labels. By fine-tuning these models on specific datasets, data scientists can create text classifiers that can automatically label text data with high accuracy;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Named Entity Recognition (NER). LLMs can be fine-tuned for NER tasks to identify and label entities such as names, dates, locations, and more in unstructured text data;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Sentiment Analysis. LLMs can determine the sentiment of a piece of text (e.g., positive, negative, neutral), which is valuable for tasks like customer reviews, social media sentiment analysis, and others;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Text Generation. In some cases, LLMs can generate labels or summaries for text data, simplifying the labeling process. For instance, you can use LLMs to generate short product descriptions in an e-commerce dataset;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Question Answering. LLMs can answer questions about text, making it possible to automatically generate labels by asking questions about the content of the data;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Language Translation. LLMs can translate text between languages, which can be useful for labeling multilingual datasets.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Integration of Large Language Models has expanded the capabilities of auto-labeling, making it a valuable tool in modern workflows. By automating a significant portion of the labeling process, LLMs enhance productivity in labeling projects, allowing organizations to meet tighter deadlines and use their resources more efficiently. This automation liberates human annotators from mundane and repetitive work, allowing them to focus on more complex and nuanced aspects of the task.\"}),/*#__PURE__*/e(\"p\",{children:\"While Large Language Models offer numerous advantages in natural language understanding and generation, they also come with their fair share of challenges.\"}),/*#__PURE__*/t(\"h3\",{children:[\"Challenges of \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/labeling-with-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"data labeling with LLMs\"})})]}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Data Biases. LLMs can inherit biases from the data they have been trained on, potentially leading to biased labels;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Limited to Text Data. LLMs are primarily designed for text data, so they may not be as effective for labeling other types of data, such as images or video;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Continuous Maintenance. LLMs require continuous monitoring and maintenance to ensure that they provide accurate and up-to-date labels, as the model's performance may degrade over time;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Overconfidence. LLMs can exhibit overconfidence in their predictions, providing labels with high certainty even when they are incorrect.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"In practice, addressing these challenges involves a hybrid approach that combines the strengths of LLMs for automated data labeling with human labelers for validation and correction. This balance helps leverage the efficiency of LLMs while ensuring the accuracy and quality of labeled data, particularly in complex or sensitive domains.\"}),/*#__PURE__*/e(\"h2\",{children:\"Hybrid Labeling: Combining Human Expertise with LLMs\"}),/*#__PURE__*/e(\"p\",{children:\"Although LLMs offer unparalleled efficiency and quality in automated data labeling, there are still scenarios where human expertise is essential. Hybrid labeling is a powerful tool that combines the strengths of both humans and LLMs.\"}),/*#__PURE__*/t(\"p\",{children:[\"Platforms like Toloka offer \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/labeling-with-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"hybrid labeling solutions\"})}),\", allowing organizations to make use of the precision of human data labeling alongside the speed and efficiency of LLMs. In this approach, LLMs create pre-labeled data, and human annotators review and refine the labels, ensuring accuracy and compliance with specific requirements.\"]}),/*#__PURE__*/e(\"p\",{children:\"In this approach, the question of who should label raw data isn't always straightforward. Toloka's approach is iterative. Data labeling isn't a one-time task, it's a continuous process of improvement. This iterative approach helps fine-tune LLMs and improves the overall quality of labeled data over time.\"}),/*#__PURE__*/t(\"p\",{children:[\"Here's how Toloka optimizes \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/labeling-with-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"data labeling pipelines with LLMs\"})}),\":\"]}),/*#__PURE__*/t(\"ol\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\",\"--list-style-type\":\"unset\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"LLM processes data and suggests labels for human annotators;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Qualified annotators step in to label edge cases and other instances that require nuanced judgment. Their domain knowledge ensures accurate labeling in complex scenarios;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Humans conduct selective evaluations of LLM-generated annotations. This step helps identify and correct any discrepancies or errors in the initial labels;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Expert annotators provide quality assurance and feedback. Their expertise ensures that the labeled data meets the highest standards of accuracy and relevance;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Toloka collaborates with domain experts who bring field-specific knowledge to the labeling process. This expertise is essential for tasks that require a deep understanding of the subject.\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Benefits of Hybrid Labeling\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"High Precision. Human annotators can handle complex or ambiguous cases, ensuring the highest level of accuracy;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Scalability. LLMs provide the initial labeling, allowing organizations to process large datasets quickly, while humans handle the final quality control;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Flexibility. Organizations can customize the level of human involvement based on the nature of the labeling task, and optimizing resources.\"})})]}),/*#__PURE__*/t(\"h3\",{children:[\"What is required for automated \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/labeling-with-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"data labeling with LLMs\"})}),\" to provide the best performance possible?\"]}),/*#__PURE__*/e(\"p\",{children:\"LLMs demonstrate remarkable performance in various natural language processing (NLP) tasks, often achieving or even surpassing human-level performance. For example, they excel in tasks like language translation, text summarization, sentiment analysis, and named entity recognition. However, their performance depends on the specific task and the quality of guidance or training they receive.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Guidance and Fine-Tuning\"})}),/*#__PURE__*/e(\"p\",{children:\"While LLMs offer speed and efficiency, they may not perform optimally straightaway. To achieve high accuracy, LLMs often require guidance in the form of fine-tuning. Fine-tuning involves training the model on a smaller, task-specific dataset to adapt it to a particular domain or labeling task, improving its performance. It is often necessary to ensure accurate automated data labeling with LLMs. Without proper guidance, LLMs may not perform optimally, and their outputs may be unreliable.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Human Oversight\"})}),/*#__PURE__*/e(\"p\",{children:\"LLMs can handle many tasks autonomously, but still human oversight is crucial. Humans can review and correct LLM-generated labels. LLMs are powerful language models, but they are not infallible. They can make errors, especially when dealing with ambiguous or complex data. Human oversight helps catch and correct these errors, ensuring the final labels are of high quality and accuracy. Reviewers can verify the correctness of LLM-generated labels by comparing them to ground truth or existing labeled data, improving the overall data quality.\"}),/*#__PURE__*/e(\"p\",{children:\"Fine-tuning and human assistance are often used in tandem. Fine-tuning prepares the LLM to be more task-specific and aligned with guidelines, while human assistance provides the critical human oversight necessary to ensure the quality, fairness, and accuracy of the labels generated by the LLM.\"}),/*#__PURE__*/t(\"h3\",{children:[\"Why \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/labeling-with-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Auto Data Labeling with LLMs\"})}),\" Needs Human Guidance\"]}),/*#__PURE__*/e(\"p\",{children:\"Labeling with Large Language Models is a powerful and efficient approach to data labeling, but it's not always a completely standalone solution. Human annotators can serve as a quality control mechanism by reviewing and validating LLM-generated labels and catching any errors or inaccuracies. There are several reasons why labeling with LLMs may benefit from human assistance:\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Handling Edge Cases\"})}),/*#__PURE__*/e(\"p\",{children:\"LLMs may struggle with rare or unusual cases that do not conform to standard patterns. Such instances are called edge cases and they often deviate from the norm and can be challenging to label accurately due to their uniqueness or complexity. Edge cases are characterized by their low frequency or unpredictability. LLMs may struggle to assign accurate labels to edge cases, as they rely on statistical patterns and may lack specific knowledge about these unique instances. Humans can handle edge cases effectively, preventing mislabeling and ensuring comprehensive coverage of the data.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Ambiguity Handling\"})}),/*#__PURE__*/e(\"p\",{children:\"LLMs may encounter situations where data is ambiguous. Many words and phrases have multiple meanings depending on the context. LLMs might select a meaning that seems most probable based on statistical patterns, but humans can infer the intended meaning more accurately by drawing on their knowledge of the subject matter. Human annotators can help disambiguate such cases, ensuring the correct label is applied.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Bias Mitigation\"})}),/*#__PURE__*/e(\"p\",{children:\"LLMs can inherit biases present in their training dataset, potentially leading to biased labels. Human guidance is essential for recognizing and correcting biased or inappropriate labeling to ensure fairness and ethical considerations. Forming diverse and inclusive annotation teams with a variety of perspectives can contribute to reducing biases. Diverse teams are more likely to catch and address biases effectively.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Ethical and Sensitive Content\"})}),/*#__PURE__*/e(\"p\",{children:\"LLMs may not always be equipped to handle content that is sensitive, controversial, or ethically challenging. Human guidance ensures that the labeling process adheres to contemporary ethical standards and sensitivity to cultural shifts. Annotators can exercise judgment and make appropriate decisions in such cases.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Adaptation to Specific Requirements\"})}),/*#__PURE__*/e(\"p\",{children:\"Some labeling tasks have unique requirements that LLMs may not fully understand. Human annotators can tailor the labeling process to meet these specific needs, ensuring the data is labeled according to the desired criteria. In some projects, particularly those involving specialized domains or historical references, human annotators with expertise in the subject matter are invaluable. They can accurately label data based on their domain knowledge, which LLMs may lack.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Evolution of Language and Context\"})}),/*#__PURE__*/e(\"p\",{children:\"Language is dynamic, and contextual nuances evolve over time. LLMs are trained on vast datasets, and their pre-training data can quickly become outdated as language trends and cultural contexts change. Human guidance is essential to bridge these gaps and ensure that labels remain contextually relevant. Feedback can be used to fine-tune LLMs over time, helping them adapt to changing language trends and evolving contexts. This iterative process of improvement through human guidance allows LLMs to remain relevant.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Slang and informal language\"})}),/*#__PURE__*/e(\"p\",{children:\"LLMs can struggle with slang and informal language. This is because slang often involves unconventional word usage, idiomatic expressions, or cultural references that may not be well-represented in their training data. Oversight and review are valuable in these cases, as at times only humans with knowledge of the specific slang can correct and provide context to ensure accurate labeling or understanding of the data. Additionally, domain-specific fine-tuning or training on specialized input data that includes slang can improve an LLM's performance in handling such language variations.\"}),/*#__PURE__*/e(\"p\",{children:\"LLMs are unquestionably powerful tools for automating data labeling, but potential issues related to its use which stem from various edge cases, informal or sensitive content, unique labeling requirements, ambiguous or outdated pre-training data, and evolving language trends underline the importance of human guidance. The combination of automation capabilities and human oversight allows to find a balance between efficiency and quality, ensuring that labels to data points are accurate, reliable, and up-to-date while adhering to ethical standards.\"}),/*#__PURE__*/e(\"p\",{children:\"Moreover, learning from edge cases, ambiguities, and other challenging instances is essential for model improvement. Human feedback on how to handle specific cases can be used to enhance LLMs and make them more robust over time.\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/e(\"p\",{children:\"Data labeling is a crucial step in training machine learning models. While labeling data manually has been a traditional method of annotation, it is becoming increasingly outdated for modern enterprises due to its limitations in scalability, cost-efficiency, accuracy, and speed. Automated approaches, especially those incorporating Large Language Models, have emerged as superior alternatives that address these shortcomings and pave the way for more efficient and effective data labeling processes in the era of artificial intelligence and machine learning.\"}),/*#__PURE__*/t(\"p\",{children:[\"LLMs bring speed, stability, and cost-efficiency to data labeling, making them the future of automated data annotation. Hybrid labeling, combining human expertise with LLMs, represents a pragmatic approach that leverages the strengths of both to achieve the highest levels of precision and scalability. Platforms like Toloka offer a seamless \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/labeling-with-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"integration of LLMs and human annotators\"})}),\", allowing organizations to unlock the full potential of data labeling.\"]})]});\nexport const __FramerMetadata__ = {\"exports\":{\"richText4\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText7\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText5\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText3\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText2\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText1\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText6\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"__FramerMetadata__\":{\"type\":\"variable\"}}}"],
  "mappings": "qSAAyS,IAAMA,EAAsBC,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAS,6EAA6E,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4UAA6T,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,sXAAyXE,EAAEC,EAAE,CAAC,KAAK,4GAA4G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,iFAA8FF,EAAEC,EAAE,CAAC,KAAK,4DAA4D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,EAAE,mFAAgGF,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6EAA6E,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,8EAA8E,UAAU,eAAe,OAAO,OAAO,IAAI,uEAAuE,OAAO,wQAAwQ,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,aAA0BE,EAAEC,EAAE,CAAC,KAAK,mEAAmE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,oaAAibF,EAAEC,EAAE,CAAC,KAAK,8CAA8C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,+EAA+E,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2mBAA4lB,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,mEAAmE,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4FAAyGE,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAE,8EAA2FF,EAAEC,EAAE,CAAC,KAAK,wFAAwF,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4CAA4C,CAAC,CAAC,CAAC,EAAE,sMAAsM,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,2HAAwIE,EAAEC,EAAE,CAAC,KAAK,+EAA+E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,2CAA2C,CAAC,CAAC,CAAC,EAAE,+GAA+G,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,sDAAsD,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,+EAA+E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,2CAA2C,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,MAAM,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,0BAA0B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wSAA2SE,EAAEC,EAAE,CAAC,KAAK,8CAA8C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yDAAoD,CAAC,CAAC,CAAC,EAAE,6CAAwC,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,uCAAuC,UAAU,eAAe,OAAO,OAAO,IAAI,qEAAqE,OAAO,6VAA6V,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,8EAA2FE,EAAEC,EAAE,CAAC,KAAK,uDAAuD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAE,6cAAmc,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,8HAA2IE,EAAE,OAAO,CAAC,SAAS,mBAAmB,CAAC,EAAE,4EAAyFA,EAAE,OAAO,CAAC,SAAS,mBAAmB,CAAC,EAAE,kEAA+EA,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,+MAA4NA,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,4FAA2EA,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,SAAsBA,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,yBAAyB,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,8EAAyE,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,8CAA8C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,2DAAsD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,OAAoBE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,sEAAmFA,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,wCAAqDA,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,gTAAwTA,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,gGAAqEA,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,yCAAsDA,EAAEC,EAAE,CAAC,KAAK,oCAAoC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,wGAAqG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,8CAA8C,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,8CAA8C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,2DAAsD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iBAA8BE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,0LAAwLA,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,EAAE,yBAAyB,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,qGAAgG,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,2CAA2C,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,8CAA8C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,2DAAsD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0EAA0E,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,8FAAyF,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,8CAA8C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,2DAAsD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+LAA+L,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gDAA2C,CAAC,EAAE,mSAA8R,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sCAAsC,CAAC,EAAE,kRAAgSA,EAAE,KAAK,CAAC,SAAS,MAAM,CAAC,EAAE,iEAAkE,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wHAAwH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iGAAiG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,0NAAmOE,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,EAAE,KAAM,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,6XAA6X,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6IAA6I,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iSAAiS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0DAAqD,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,kBAAkB,UAAU,eAAe,OAAO,OAAO,IAAI,sEAAsE,OAAO,kWAAkW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,WAAwBE,EAAEC,EAAE,CAAC,KAAK,8EAA8E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iDAAiD,CAAC,CAAC,CAAC,EAAE,0QAAqQ,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,oBAAoB,UAAU,eAAe,OAAO,OAAO,IAAI,sEAAsE,OAAO,kWAAkW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gcAAwcE,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,oBAAoB,UAAU,eAAe,OAAO,OAAO,IAAI,wEAAwE,OAAO,0WAA0W,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gIAA6IE,EAAEC,EAAE,CAAC,KAAK,yEAAyE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,EAAE,aAAa,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iIAA8IE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yCAAyC,CAAC,CAAC,CAAC,EAAE,2CAAwDF,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAE,wDAAmD,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,gJAAmJE,EAAEC,EAAE,CAAC,KAAK,kCAAkC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wDAAwD,CAAC,CAAC,CAAC,EAAE,OAAoBF,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yCAAyC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,2DAA2D,CAAC,CAAC,CAAC,CAAC,EAAeG,EAAuBL,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,qEAAwEE,EAAEC,EAAE,CAAC,KAAK,yDAAyD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4BAA4B,CAAC,CAAC,CAAC,EAAE,qOAA2N,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,+FAA0F,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oWAA+V,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,oGAA4GE,EAAEC,EAAE,CAAC,KAAK,gHAAgH,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,6KAA6K,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yNAAyN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kVAA6U,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,aAAa,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sMAAsM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6PAA6P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,OAAO,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,EAAE,6BAA6B,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,UAAU,CAAC,EAAE,wBAAwB,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,UAAU,CAAC,EAAE,iCAAiC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,EAAE,kBAAkB,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yPAAyP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0OAA0O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iOAAiO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yQAA+P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sHAAsH,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yKAAyK,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2JAAsJ,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wIAAwI,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4IAA4I,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2HAA2H,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6JAA6J,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,iBAAiB,YAAY,YAAY,YAAY,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEI,EAAE,CAAC,oBAAoB,sEAAsE,SAASC,GAAgBL,EAAEM,EAAE,CAAC,GAAGD,EAAE,KAAK,MAAM,WAAW,GAAG,UAAU,eAAe,IAAI,8BAA8B,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeL,EAAE,IAAI,CAAC,SAAS,qLAAgL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kIAAkI,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uGAAkG,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qCAAqC,CAAC,EAAE,oNAAoN,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sDAAsD,CAAC,EAAE,sPAAsP,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,+BAA+B,CAAC,EAAE,4MAA0MA,EAAEC,EAAE,CAAC,KAAK,8BAA8B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,4FAA4F,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,+LAAkME,EAAEC,EAAE,CAAC,KAAK,6CAA6C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,8BAAyB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeK,EAAuBT,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6NAAwN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wPAAwP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wUAAmU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wZAA8Y,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wBAAmB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6bAA6b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mCAAmC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wGAAwG,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iHAAiH,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kJAAkJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4MAA4M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+UAA0U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oQAA+P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yDAAyD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oEAA+D,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeQ,EAAuBV,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,8XAA8X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wZAAwZ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8BAA8B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sdAAsd,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,udAAud,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uUAAuU,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+FAA+F,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sEAAsE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gGAAgG,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sFAAsF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yIAAyI,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gVAAgV,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yUAAyU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4TAA4T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0OAA0O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4YAA4Y,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uCAAuC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4DAA4D,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yCAAyC,CAAC,EAAE,2VAA2V,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,sMAAsM,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,8OAA8O,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,kNAAkN,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,gNAAgN,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,uLAAuL,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iFAAiF,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6eAA6e,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oGAA+F,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8NAA8N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+IAA+I,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8CAA8C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4DAA4D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+DAA+D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mDAAmD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mEAAmE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yiBAAyiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qTAAqT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wVAAwV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qQAAqQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oRAAoR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sVAAsV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,8CAA8C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iZAAiZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mXAAmX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gVAAgV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2LAA2L,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sHAAsH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kHAAkH,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qVAAqV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wTAAwT,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yLAAyL,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0SAA0S,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8WAA8W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yXAAoX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4WAA4W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mVAAmV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gfAAgf,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0TAA0T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+dAA+d,CAAC,CAAC,CAAC,CAAC,EAAeS,EAAuBX,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,ybAAyb,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,qIAAqI,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,UAAU,eAAe,kBAAkB,MAAM,SAAsBF,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,oQAAoQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+EAA+E,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,geAAge,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4HAA4H,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uQAAuQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2PAA2P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wWAAwW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4PAA4P,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ucAAuc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yZAAyZ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kUAAkU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yYAAyY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uWAAuW,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gBAA6BE,EAAEC,EAAE,CAAC,KAAK,qGAAqG,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,qaAAqa,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,yQAAyQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wZAAwZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8NAA8N,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sdAAsd,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8TAA8T,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gcAAgc,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wWAAwW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAyB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+PAA+P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oPAAoP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qiBAAgiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0ZAA0Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0NAA0N,CAAC,CAAC,CAAC,CAAC,EAAeU,EAAuBZ,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,sdAAsd,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,uBAAoCE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAE,yMAAyM,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6cAA6c,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6bAA6b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wWAAwW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sYAAsY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kXAA6W,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oWAAoW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ycAAyc,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,klBAAklB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,skBAAikB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gXAAgX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ihBAAihB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4XAA4X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+WAA+W,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,iKAAiK,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,4JAA4J,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,EAAE,oJAAoJ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,EAAE,sRAAsR,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,+LAA+L,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+XAA+X,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oKAAoK,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,6HAA6H,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,EAAE,qQAAkRA,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,wBAAwB,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,yLAAyL,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,eAAe,CAAC,EAAE,6JAA6J,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAoX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wNAAwN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qCAAqC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+EAA4FE,EAAEC,EAAE,CAAC,KAAK,2HAA2H,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,wRAAwR,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,OAAO,oBAAoB,OAAO,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,ubAAub,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,EAAE,+qBAAirB,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+gBAA+gB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4WAA4W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4UAA4U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6ZAA6Z,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sDAAsD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qdAAqd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oVAAoV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oUAAoU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2aAA2a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kZAAkZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yiBAAyiB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uXAAuX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gBAAgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gXAAgX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2YAA2Y,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+WAA+W,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mcAAmc,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0cAA0c,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ySAAyS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sdAAsd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAA6T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kRAAkR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mfAAmf,CAAC,CAAC,CAAC,CAAC,EAAeW,EAAuBb,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,yXAAsYE,EAAEC,EAAE,CAAC,KAAK,mIAAmI,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,gMAAgM,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,qWAAqW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0WAA0W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kXAAkX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8BAA8B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8hBAAyhB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2QAA2Q,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mYAA8X,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,OAAoBE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,EAAE,waAAwa,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0VAA0V,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,0KAAuLE,EAAEC,EAAE,CAAC,KAAK,mDAAmD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAE,sgBAAwgB,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0cAA0c,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,6CAA6C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,8WAAyW,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAAsP,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,yBAAsCE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,0aAAkbF,EAAEC,EAAE,CAAC,KAAK,2EAA2E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,MAAM,CAAC,CAAC,CAAC,EAAE,4FAA4F,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oPAAoP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sgBAA6e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qIAAqI,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qGAA6GE,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,0BAA0B,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeU,EAAuBd,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,oUAAoU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oUAAoU,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qUAAqU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8PAA8P,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6PAA6P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4aAA4a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yRAAyR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sdAAsd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0TAA0T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0VAA0V,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oDAAoD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oZAAoZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,saAAsa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iaAAia,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6MAA6M,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,6QAA6Q,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,kPAAkP,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,eAAe,CAAC,EAAE,qRAAqR,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8iBAA8iB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mYAAmY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qDAAqD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4eAA4e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2NAA2N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gUAAgU,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,EAAE,qNAAqN,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAC,qBAAkCE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,mCAAmC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2JAA2J,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+JAA+J,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uLAAuL,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uEAAuE,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4BAAyCE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAE,mBAAmB,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6PAA6P,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0KAA0K,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8MAA8M,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wNAAwN,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kKAAkK,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0HAA0H,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,keAAke,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6JAA6J,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,iBAA8BE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qHAAqH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6JAA6J,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0LAA0L,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0IAA0I,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kVAAkV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sDAAsD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2OAA2O,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+BAA4CE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,2BAA2B,CAAC,CAAC,CAAC,EAAE,0RAA0R,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,mTAAmT,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+BAA4CE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,mCAAmC,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,OAAO,oBAAoB,OAAO,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8DAA8D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4KAA4K,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4JAA4J,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gKAAgK,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6LAA6L,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iHAAiH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0JAA0J,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6IAA6I,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,kCAA+CE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,EAAE,4CAA4C,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,yYAAyY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6eAA6e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iiBAAiiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wSAAwS,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,OAAoBE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,EAAE,uBAAuB,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,0XAA0X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6kBAA6kB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6ZAA6Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qaAAqa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,+BAA+B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAA6T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,qCAAqC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ydAAyd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,mCAAmC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sgBAAsgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,glBAAglB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yiBAAyiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sOAAsO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ijBAAijB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,yVAAsWE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,0CAA0C,CAAC,CAAC,CAAC,EAAE,yEAAyE,CAAC,CAAC,CAAC,CAAC,CAAC,EAC3hkIW,EAAqB,CAAC,QAAU,CAAC,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,SAAW,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,mBAAqB,CAAC,KAAO,UAAU,CAAC,CAAC",
  "names": ["richText", "u", "x", "p", "Link", "motion", "richText1", "ComponentPresetsConsumer", "t", "Youtube", "richText2", "richText3", "richText4", "richText5", "richText6", "richText7", "__FramerMetadata__"]
}