{
  "version": 3,
  "sources": ["ssg:https://framerusercontent.com/modules/YVgynkVt96a0ES9Gk8yz/e2OiwG21hMIpRxQ1FbTF/eo4RAmtig-22.js"],
  "sourcesContent": ["import{jsx as e,jsxs as t}from\"react/jsx-runtime\";import{Link as a}from\"framer\";import{motion as n}from\"framer-motion\";import*as o from\"react\";export const richText=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"If your business relies on user-generated content, you're probably already using some type of content moderation. But are you sure you have the best moderation tool for your product and your bottom line? \"}),/*#__PURE__*/e(\"p\",{children:\"Content moderation can have a hidden impact on your business. Let's look at how you can optimize your moderation process for better product metrics.\"}),/*#__PURE__*/e(\"h2\",{children:\"Products that benefit from moderation\"}),/*#__PURE__*/e(\"p\",{children:\"Content moderation isn't just for social media. Here are some typical use cases where moderation plays a role in product success.\"}),/*#__PURE__*/e(\"img\",{alt:\"streaming services\",className:\"framer-image\",height:\"40\",src:\"https://framerusercontent.com/images/mHr5WhksOfWAc1QmbUg1YN8a7A.png\",srcSet:\"https://framerusercontent.com/images/mHr5WhksOfWAc1QmbUg1YN8a7A.png?scale-down-to=512 512w,https://framerusercontent.com/images/mHr5WhksOfWAc1QmbUg1YN8a7A.png 516w\",style:{aspectRatio:\"516 / 80\"},width:\"258\"}),/*#__PURE__*/e(\"p\",{children:\"New features let viewers share their opinions and leave reviews on the shows they've watched, allowing online streaming platforms to benefit from increased user engagement. But sometimes anonymity brings out the worst in people. You need fast, reliable moderation to prevent a couple of toxic users from ruining the experience for everyone.\"}),/*#__PURE__*/e(\"img\",{alt:\"video games\",className:\"framer-image\",height:\"32\",src:\"https://framerusercontent.com/images/VSw6neoLEipAkHHVRDZ97gw6ZI.png\",style:{aspectRatio:\"382 / 64\"},width:\"191\"}),/*#__PURE__*/e(\"p\",{children:\"One of the biggest problems of the gaming industry is toxic content in multiplayer environments. Socializing is an important part of the game for most players, but voice chat is especially hard to monitor for toxic language and insults. The negative behavior of a few players can discourage others from playing. Protect customers on your gaming platform by moderating player communication as it happens. \"}),/*#__PURE__*/e(\"img\",{alt:\"datings apps\",className:\"framer-image\",height:\"35\",src:\"https://framerusercontent.com/images/q8JbcLwipAgAgUxzjmpSbogftN8.png\",style:{aspectRatio:\"358 / 70\"},width:\"179\"}),/*#__PURE__*/e(\"p\",{children:\"Even a single toxic encounter can make a user leave a dating platform. Detecting and preventing bad behavior by removing offenders can make the community healthier, ultimately improving metrics with better overall user retention.\"}),/*#__PURE__*/e(\"img\",{alt:\"company blogs\",className:\"framer-image\",height:\"36\",src:\"https://framerusercontent.com/images/IdSVuTFWYfsl2G7VTStN2htbA.png\",style:{aspectRatio:\"432 / 72\"},width:\"216\"}),/*#__PURE__*/e(\"p\",{children:\"Sometimes you need to watch for problems more subtle than toxic content. For instance, if you allow comments on a company blog or specialized forum, you don't want people praising your competitors. Regular moderation services don't cover cases like this, so you need a custom model to detect specific content. You can run the custom model on your website for pre-moderation, meaning comments won't be posted without \u201Capproval\u201D, or post-moderation, meaning irrelevant comments will be deleted soon after posting. \"}),/*#__PURE__*/e(\"img\",{alt:\"e-commerce marketplace\",className:\"framer-image\",height:\"35\",src:\"https://framerusercontent.com/images/Spimqk8WDl7t29vQt3oFPB2cdmA.png\",srcSet:\"https://framerusercontent.com/images/Spimqk8WDl7t29vQt3oFPB2cdmA.png?scale-down-to=512 512w,https://framerusercontent.com/images/Spimqk8WDl7t29vQt3oFPB2cdmA.png 670w\",style:{aspectRatio:\"670 / 70\"},width:\"335\"}),/*#__PURE__*/e(\"p\",{children:\"Modern e-commerce websites are mostly user-generated, from product pages to reviews and comments. All text, images, and videos need to pass moderation across multiple categories to prevent liability for anything offensive, inappropriate, or illegal on the site.\"}),/*#__PURE__*/e(\"h2\",{children:\"Typical solutions for analyzing content\"}),/*#__PURE__*/e(\"p\",{children:\"There are three ways to protect your platform from undesirable content: hire a team of moderators, use automated moderation tools or services, or combine automated tools with manual moderation on edge cases.\"}),/*#__PURE__*/e(\"p\",{children:\"Hiring an in-house moderation team isn't efficient for several reasons:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"The workload is unpredictable, and even a large group of moderators can't always handle spikes in activity.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Hiring can be expensive, especially if you consider the costs of employee benefits and people management.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Human moderation tends to be inconsistent because people interpret and evaluate content subjectively.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Manual moderation doesn't work in real time because humans need time to analyze content. And if your platform has millions of users, it's virtually impossible to moderate millions of comments manually.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Illegal content shouldn't be shown to human moderators because your platform could be held liable for distribution.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Creating your own ML-based solution is even more costly, especially if you need to monitor both texts and images, or even videos and voice chats. Text moderation is straightforward because models are pre-trained to detect certain words and expressions. Images, voice, and videos are much more challenging, and only state-of-the-art models can recognize patterns in visual content.\"}),/*#__PURE__*/e(\"p\",{children:\"This challenge deters many product teams from introducing collaboration features, while their competitors race ahead.\"}),/*#__PURE__*/e(\"h2\",{children:\"Effective, timely moderation at lower costs\"}),/*#__PURE__*/e(\"p\",{children:\"Toloka Moderation automates the process with an AI model tuned to your needs. Instead of employing a team of moderators and managers, our clients usually keep just 2 or 3 moderators on board to check the model's performance and run quality control. The overall cost of automation plus quality control is much lower than the average spending on manual moderation on a large scale.\"}),/*#__PURE__*/e(\"p\",{children:\"An additional advantage is the opportunity for multi-language moderation. If your business is a startup operating in 15 languages, hiring a team of moderation experts for every language could be cost-prohibitive, especially for grammatically complex languages like Arabic that tend to cost more. With Toloka you get an automated solution with the skills of a multilingual team for the price of a single moderator.\"}),/*#__PURE__*/e(\"p\",{children:\"In terms of ethics, automated solutions do a better job of protecting people from harmful content. Global content sharing platforms usually have limited contracts with moderators for the span of a few months. The purpose of short-term contracts is to protect workers from the effects of repeated exposure to violent or toxic content. Using an AI-based solution is a more ethical way to prevent harm.\"}),/*#__PURE__*/e(\"h3\",{children:\"Manual moderation vs AI solutions\"}),/*#__PURE__*/e(\"img\",{alt:\"Manual moderation vs AI solutions\",className:\"framer-image\",height:\"590\",src:\"https://framerusercontent.com/images/rZXliPZmrobB63T1WNPYQIq2rYs.png\",srcSet:\"https://framerusercontent.com/images/rZXliPZmrobB63T1WNPYQIq2rYs.png?scale-down-to=512 512w,https://framerusercontent.com/images/rZXliPZmrobB63T1WNPYQIq2rYs.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/rZXliPZmrobB63T1WNPYQIq2rYs.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/rZXliPZmrobB63T1WNPYQIq2rYs.png 2594w\",style:{aspectRatio:\"2594 / 1180\"},width:\"1297\"}),/*#__PURE__*/e(\"p\",{children:\"Some content categories require special approaches. For instance, dating apps have prevalent problems with illegal activity like blackmail or secret money transfers. These cases aren't detected by standard moderation services. Toloka Moderation can handle any custom classes with minimal training time for fast deployment. \"}),/*#__PURE__*/e(\"h2\",{children:\"The impact on business metrics\"}),/*#__PURE__*/e(\"p\",{children:\"Each of the business cases above illustrates how companies introduce user-generated content in their products and potentially grow the retention rate. Why? Because collaboration between users always creates additional value.\"}),/*#__PURE__*/e(\"p\",{children:\"All the important digital metrics are about customer engagement. Retention is a key metric \u2014 successful products always keep customers on the platform long-term. But frequency of use is also important.\"}),/*#__PURE__*/e(\"p\",{children:\"Retention and frequency both contribute to lifetime value, which reflects how much money a user brings to the platform during the retention period. For instance, let's say we spend $10 on customer acquisition for a user who generates $7 of monthly revenue for 3 months, giving us $11 of profit (we can ignore operational costs in this equation because they are almost constant). If we can make positive changes to moderation that increase average retention to 4 months, we earn $18 profit per user, which is a 64% growth in lifetime value.\"}),/*#__PURE__*/e(\"p\",{children:\"Multiplied by thousands of users, small improvements can make a huge difference!\"}),/*#__PURE__*/e(\"p\",{children:\"Grow your business and achieve the metrics it deserves. Get automated moderation with human insight by Toloka:\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(a,{href:\"https://toloka.ai/moderation/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Learn more\"})})})]});export const richText1=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Large language models are the most powerful AI solutions available. With great power comes great responsibility, and model producers rely on fine-tuning to make model output accurate and beneficial to end users. Human alignment can be a vital part of fine-tuning \u2014 a strong antidote to LLM flaws like hallucination and bias. \"}),/*#__PURE__*/t(\"p\",{children:[\"Toloka is building datasets of human prompts and completions for fine-tuning and evaluating LLMs. The \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/creative-writing-prompts-for-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"prompts come from a diverse crowd\"})}),\" of thousands of Tolokers, representing 69 countries. But the completions for these prompts are written and carefully reviewed by curated teams of experts with specialized skills in writing, editing, fact-checking, and domain-specific knowledge. They focus on crafting thorough responses in a collaborative workflow that promotes quality writing.\"]}),/*#__PURE__*/e(\"p\",{children:\"Who are the experts, and how do we leverage their diverse set of skills? To answer these questions, we talked to several AI Tutors on our writing team. We'll share their insights into the process of writing for generative AI.\"}),/*#__PURE__*/e(\"h2\",{children:\"Why LLMs need writing experts\"}),/*#__PURE__*/e(\"p\",{children:\"Generative language models learn language almost like children do, through exposure to a vast amount of language input. These models don't need us to teach them grammar and vocabulary because they master the language during pre-training. The next step is fine-tuning, where they learn what good texts look like and gain in-depth knowledge in specific areas, more like sending a student to high school or college.\"}),/*#__PURE__*/e(\"p\",{children:\"This is why we call our experts AI Tutors \u2014 they share their skills with the model to fill in the gaps, making the model's output more sophisticated and refined. As domain experts and professional writers, they offer contextual understanding and a discerning eye to improve the data used for tuning LLMs.\"}),/*#__PURE__*/e(\"p\",{children:\"Feyaza, one of the editors on our AI Tutors team, sees human experts as playing the role of the playwriter directing AI. \u201CI think even if AI were to be the original source for the canonical copy of that play or book or whatever, there's always a consultant role for humans,\u201D she muses.\"}),/*#__PURE__*/e(\"h2\",{children:\"Up close and personal: meet 3 of our experts\"}),/*#__PURE__*/e(\"p\",{children:\"The stars of today's post are Feyaza, Luis, and Nicole, writers and editors on the AI Tutors team. We learned about their backgrounds and what it's like writing for AI.\"}),/*#__PURE__*/e(\"img\",{alt:\"Up close and personal: meet 3 of our experts\",className:\"framer-image\",height:\"653\",src:\"https://framerusercontent.com/images/2cbvCJJM4IJWSiPr6GNVf0dVgDw.png\",srcSet:\"https://framerusercontent.com/images/2cbvCJJM4IJWSiPr6GNVf0dVgDw.png?scale-down-to=512 512w,https://framerusercontent.com/images/2cbvCJJM4IJWSiPr6GNVf0dVgDw.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/2cbvCJJM4IJWSiPr6GNVf0dVgDw.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/2cbvCJJM4IJWSiPr6GNVf0dVgDw.png 3202w\",style:{aspectRatio:\"3202 / 1306\"},width:\"1601\"}),/*#__PURE__*/e(\"img\",{alt:\"Up close and personal: meet 3 of our experts 2\",className:\"framer-image\",height:\"296\",src:\"https://framerusercontent.com/images/CP5RvK8LVrWQZFvD5YJpbe1fJPY.png\",srcSet:\"https://framerusercontent.com/images/CP5RvK8LVrWQZFvD5YJpbe1fJPY.png?scale-down-to=512 512w,https://framerusercontent.com/images/CP5RvK8LVrWQZFvD5YJpbe1fJPY.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/CP5RvK8LVrWQZFvD5YJpbe1fJPY.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/CP5RvK8LVrWQZFvD5YJpbe1fJPY.png 3208w\",style:{aspectRatio:\"3208 / 592\"},width:\"1604\"}),/*#__PURE__*/e(\"h2\",{children:\"Getting experts on board\"}),/*#__PURE__*/e(\"p\",{children:\"Each expert has to take rigorous writing tests before they can accept tasks. If they pass, they complete onboarding and training to learn how and what to write for training generative AI. We provide detailed guidelines so they know exactly what makes a good response.\"}),/*#__PURE__*/e(\"p\",{children:'One of the main jobs of AI Tutors is to write responses to prompts (called prompt completions). When they open a task, they see guidelines about the voice they should write in and other parameters. For example, the task guidelines might say \"Imagine you are a helpful AI assistant.\" The next part of the task gives them a question to answer, or sometimes an AI response to check and verify.'}),/*#__PURE__*/e(\"img\",{alt:\"Classification task\",className:\"framer-image\",height:\"540\",src:\"https://framerusercontent.com/images/qM4zfgSgKAqa6YzFM1DAOei9Pc.webp\",srcSet:\"https://framerusercontent.com/images/qM4zfgSgKAqa6YzFM1DAOei9Pc.webp?scale-down-to=512 512w,https://framerusercontent.com/images/qM4zfgSgKAqa6YzFM1DAOei9Pc.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/qM4zfgSgKAqa6YzFM1DAOei9Pc.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/qM4zfgSgKAqa6YzFM1DAOei9Pc.webp 2560w\",style:{aspectRatio:\"2560 / 1081\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Classification task\"})}),/*#__PURE__*/e(\"img\",{alt:\"Summarization task\",className:\"framer-image\",height:\"433\",src:\"https://framerusercontent.com/images/tj0CtDlaKCQtDgoHS3gSMXeDGzY.webp\",srcSet:\"https://framerusercontent.com/images/tj0CtDlaKCQtDgoHS3gSMXeDGzY.webp?scale-down-to=512 512w,https://framerusercontent.com/images/tj0CtDlaKCQtDgoHS3gSMXeDGzY.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/tj0CtDlaKCQtDgoHS3gSMXeDGzY.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/tj0CtDlaKCQtDgoHS3gSMXeDGzY.webp 2560w\",style:{aspectRatio:\"2560 / 866\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Summarization task\"})}),/*#__PURE__*/e(\"img\",{alt:\"Paraphrasing task\",className:\"framer-image\",height:\"547\",src:\"https://framerusercontent.com/images/cphJW2rsxChm7nwkh3QZl4BjGk.webp\",srcSet:\"https://framerusercontent.com/images/cphJW2rsxChm7nwkh3QZl4BjGk.webp?scale-down-to=512 512w,https://framerusercontent.com/images/cphJW2rsxChm7nwkh3QZl4BjGk.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/cphJW2rsxChm7nwkh3QZl4BjGk.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/cphJW2rsxChm7nwkh3QZl4BjGk.webp 2560w\",style:{aspectRatio:\"2560 / 1095\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Paraphrasing task \"})}),/*#__PURE__*/e(\"p\",{children:\"Writing skills are essential, but not enough to do the job right. It's like having the fuel but no car \u2014 you can't get to where you want to go. The missing element is fact-checking skills, which is why we test and train experts on their ability to track down and verify information online. You can find just about anything published on the internet, both true and untrue, and experts have to separate fact from fiction.\"}),/*#__PURE__*/e(\"p\",{children:\"Once they start writing, the experts get continuous feedback from their team leads so they are improving the quality of their responses. They can also collaborate with the team when they encounter a challenging prompt and need ideas for how to approach the task.\"}),/*#__PURE__*/e(\"h2\",{children:\"How the writing team guarantees quality\"}),/*#__PURE__*/e(\"p\",{children:\"AI writing is challenging work. Luis explains, \u201CSpeed and accuracy are key. Not only do we need to come up with a quality piece in less than an hour, but it also has to follow the rules and writing guidelines. These guidelines are in place so that the bank of texts we are creating can be a good training set for an ideal language model.\u201D\"}),/*#__PURE__*/e(\"p\",{children:\"Every single piece of writing is checked by a team lead. They make sure to follow the style guidelines and coach writers on the preferred tone of voice. Their job is to make sure that everyone understands what is good writing, and what is a little off. When the team leads have questions about difficult cases, they consult with the editor-in-chief and other leads. The team also does spot checks of the final results to make sure the overall quality is consistent.\"}),/*#__PURE__*/e(\"p\",{children:\"The writers are assigned to teams of about 10 people so they can work closely together. As a team lead, Feyaza says, \u201COne of the most challenging things is trying to be as supportive as you can for the team because everyone is in different time zones and everyone needs different things. We have people who are older, and we have people who are very young and this is their first job. So it's quite a nice mix, but you have to be available for them in every capacity that they need. You've got to really bring your A-game.\u201D\"}),/*#__PURE__*/t(\"p\",{children:[\"Manual checking is the most reliable way to control the quality of unique writing tasks, but we also calculate some automated metrics to check results before we feed them to the model. For example, we check readability, and we run \",/*#__PURE__*/e(a,{href:\"https://tatsu-lab.github.io/alpaca_eval/#Alpacaeval%20Limitations\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"AlpacaEval\"})}),\" to compare the expert answers to output from an LLM. This approach has limitations, but it gives us a quick estimation of overall quality in the context of modern LLMs.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Good texts are ethical texts\"}),/*#__PURE__*/t(\"p\",{children:[\"Similar to the \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/pdf/2203.02155.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"ChatGPT training process\"})}),\", our AI Tutors are trained to focus on 3 criteria in addition to language and writing style: every text should be helpful, honest, and harmless.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Helpful\"}),\" means that the response directly answers the question, and \",/*#__PURE__*/e(\"em\",{children:\"honest\"}),\" means that it's accurate and comprehensive. Tutors provide reliable sources for every answer they write, so fact-checking is built into the process. We also guide tutors to give objective responses to subjective prompts.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Harmless\"}),\" is a trickier category to pin down. Every response must be ethical, meaning it can't offend anyone or lead to negative consequences, either directly or indirectly. For instance, a prompt that stood out for Nicole was a task to write an email impersonating a colleague. Her response explained that the request was dishonest and unethical.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Luis shared how he learned to craft careful responses:\",/*#__PURE__*/e(\"br\",{}),\"\u201CI once was tasked to write a response to a prompt asking for a weight loss workout plan. What I failed to realize was the goal of the prompter was to lose a lot of weight in a very short time, which is very dangerous. Sometimes consequences are not obvious and the dangers are not always clear. It can take a lot of nuanced decision-making.\u201D\"]}),/*#__PURE__*/e(\"p\",{children:\"The AI Tutor training process goes a step beyond general safety by focusing on sensitive topics like politics, medicine, financial advice, religion, gender identity, discrimination, and others. For each of these topics, AI Tutors are coached to write a balanced response and advise the reader to consult a professional. With practice and feedback, they learn to spot ethical issues and tag them as \u201Csensitive\u201D in the dataset.\"}),/*#__PURE__*/e(\"h2\",{children:\"What makes AI writers tick?\"}),/*#__PURE__*/e(\"p\",{children:\"For most of our experts, it's more than just a way to earn money remotely. They find purpose and meaning through their role in developing generative AI.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Luis:\"}),\" No one complained when the wheel was invented. It allowed us to become more productive and freed up time for us to tackle bigger goals and more creative pursuits. AI will have the same effect on our society. As an AI tutor, we are helping invent the wheel. In the decades to come, LLMs will become as common as the wheels on a car. This team, and people like us, are making sure that the wheel works and is as polished as it can be.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Nicole:\"}),\" I hope that I am improving tools for human creatives and contributing in a meaningful way that will help people. This role also helps me to enhance my writing and editing.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Feyaza:\"}),\" I just want to be in the AI world, you know? Just to see where this journey goes. Also, I'm learning new things every day. It's constantly refreshing my brain. I'm a writer, and what I love about this is that it hones my skills and makes them really, really sharp.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Why is human input so valuable for fine-tuning LLMs?\"}),/*#__PURE__*/e(\"p\",{children:\"An important piece of the quality puzzle is making sure that writers are motivated and know the value of the effort they put out. We asked our experts to explain why their work is important.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Luis:\"}),\" The human brain is exceptionally good at nuanced decision-making, factoring in context, intention, and consequences. If somebody asks me about local politics, I adjust my delivery based on who I'm talking to (a neighbor or a tourist) and the context, like if I was asked during a job interview or over drinks. AI can only make limited calculations of these factors, but people have a sixth sense about these things.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Nicole:\"}),\" If the goal is to produce human-like thinking and responses, then humans should be providing the necessary information and control for that. Moreover, there are problems that are ethical in nature that can be solved by human oversight. Humans can ensure that AI learns in an unbiased manner.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Feyaza:\"}),\" There was a recent prompt that I thought was hilarious, and it said, 'Why do we sometimes find dolls scary?' The answer is that our brain is wired to look at a face within 14 milliseconds or something. And you sort of recognize the face and then you look for a mind, a person to connect to. And that's exactly how I feel about our AI writing tasks. Because if we do not feel connected to these prompts, then we won't believe them. And if we don't believe them, we don't trust them, and then they're not credible. So I think humans have to be part of the process. \"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(a,{href:\"https://toloka.ai/large-language-models/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Get expert data for fine-tuning\"})})}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Read more on building LLMs with Toloka:\"}),/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/tech-spec-for-your-own-llm/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Tech spec for your own LLM\"})}),/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/creative-writing-prompts-for-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Diversity first: how we craft creative writing prompts for fine-tuning GenAI\"})}),/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/llm-leaderboard/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Toloka's new LLM Leaderboard: Finding the best model for your business\"})})]})]});export const richText2=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"As AI advances across industries, LLMs remain key catalysts of its progress. Practitioners are developing countless LLMs for various applications in various domains. But with so many models available, how do we choose the right one for our use case?\"}),/*#__PURE__*/t(\"p\",{children:[\"The answer is \",/*#__PURE__*/e(\"strong\",{children:\"LLM leaderboards.\"})]}),/*#__PURE__*/e(\"h2\",{children:\"What is an LLM leaderboard?\"}),/*#__PURE__*/e(\"p\",{children:\"An LLM leaderboard is a ranking platform that compares the performances of different large language models. It is a contest-like environment where models are ranked for tasks like code generation, question answering, summarization, and general understanding.\"}),/*#__PURE__*/e(\"p\",{children:\"LLM leaderboards are essential because they aim to provide unbiased and uncontaminated results by comparing models side-by-side using predefined criteria. These leaderboards help users and developers learn which model is right for them through transparent and objective model evaluations.\"}),/*#__PURE__*/e(\"p\",{children:\"There are different types of leaderboards. Some assess general LLM skills, such as how relevant and coherent their responses are and whether they follow instructions. Others focus on domain-specific LLMs and their performance in specialized tasks pertinent to the field. Several platforms prioritize the practical use of LLMs and rank them based on speed and cost.\"}),/*#__PURE__*/e(\"p\",{children:\"Leaderboards also address issues like hallucinations, bias, and toxicity, much-needed factors for ensuring responsible use of LLMs.\"}),/*#__PURE__*/e(\"p\",{children:\"This article explores how leaderboards test and rank models, along with examples of the most popular ones.\"}),/*#__PURE__*/e(\"h2\",{children:\"How to decide which model is the best?\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Leaderboards use different methods to grade LLMs, which generally include:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:'Quantitative metrics, usually accompanied by a minimum threshold that defines if a model is \"good\" at the task in hand or,'})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"A relative comparison against other models using human evaluation.\\xa0\"})})]}),/*#__PURE__*/e(\"p\",{children:\"A leaderboard tests the models using standardized criteria. Comprehensive evaluations ensure a fair comparison and help users learn which models achieve higher accuracies, lower costs, and turbo speeds. The criteria can vary from task to task based on required capabilities, price point, scale, and field of use. Most leaderboards include metrics that analyze these aspects in some way. Let's discuss some of the most common ones.\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Accuracy\"}),\" is a simple but key concept. In tasks like classification, we want to know if the model produces the correct outputs and use accuracy to measure its success. \",/*#__PURE__*/e(\"strong\",{children:\"F1 score\"}),\" is often used as an alternative to accuracy. It tells us how well the model balances precision and recall, which is especially valuable in imbalanced classification problems.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Speed-related metrics\"}),\" are crucial for real-time applications like chat. Therefore, model ranking platforms usually include data on how quickly a model responds or how many tokens it generates per second.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[\"Then there's \",/*#__PURE__*/e(\"strong\",{children:\"perplexity\"}),\", a score that shows a model's predictive capabilities. A lower perplexity score means the model is better at generating coherent and logically structured text, essential for tasks like text generation, machine translation, and summarization.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[\"Other metrics measure contextual relevance and semantic understanding. For example, the \",/*#__PURE__*/e(\"strong\",{children:\"BERT score\"}),\" indicates how accurately the model represents the semantics of a reference text. Such evaluations are commonly used in text summarization or translation tasks.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Ethical and responsible AI is getting more attention and requires metrics addressing this aspect of LLMs. These metrics are crucial for real-world applications to ensure the model complies with law regulations and won't provide unethical content.\\xa0\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[\"Specific leaderboards also evaluate a \",/*#__PURE__*/e(\"strong\",{children:\"model's likelihood of hallucinating\"}),\" and whether its output is correct and based on a verifiable source.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"These are some frequently used metrics, often combined in an overall score of a model's abilities.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"The choice of metrics depends on what we seek in a model. Still, regardless of the use case, metrics must accurately assess the required capabilities of the LLMs while still being:\\xa0\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Simple enough to clearly understand what they measure,\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Reliable to ensure consistency and invariant outcomes and\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Relevant to the model's purpose.\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"How do we make sure the results are objective?\"}),/*#__PURE__*/e(\"p\",{children:\"As many LLMs differ in nuances, selecting the best model can be challenging.\"}),/*#__PURE__*/e(\"p\",{children:\"To minimize bias and assure fairness, we evaluate the models against designated standards - benchmarks. These consist of problems, questions, and tasks with ground-truth answers. The LLMs' outputs are scored based on their similarity to the expected answer. Depending on the benchmark, we can assess the model's proficiency in code generation, problem-solving, and even emotional intelligence. Benchmarks are essential for fair comparison because they define clear and consistent rules for measuring performance, scale, and capabilities. Some benchmarks have established their place in the community for their holistic approach to measuring LLM performance. Let's look at some of the most popular benchmarks at the moment.\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://eqbench.com/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"EQ-Bench\"})}),\" tests the emotional intelligence of LLMs, which is still relatively overlooked on other platforms. This benchmark evaluates the models' ability to interpret social interactions by asking them how intense emotions are in different dialogues. The benchmark even strongly connects to other multi-domain datasets like\",/*#__PURE__*/e(a,{href:\"https://paperswithcode.com/dataset/mmlu\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" MMLU\"})}),\", suggesting it evaluates similar aspects of general intelligence.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://huggingface.co/datasets/openai/openai_humaneval\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"HumanEval\"})}),\" is another well-known benchmark that assesses how accurately models generate code from docstrings. Practitioners utilize it to determine a model's programming knowledge and general approach to problem-solving.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[\"Measuring Massive Multitask Language Understanding (\",/*#__PURE__*/e(a,{href:\"https://paperswithcode.com/dataset/mmlu\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"MMLU\"})}),\") is one of the most comprehensive benchmarks for LLM evaluations.\",/*#__PURE__*/e(a,{href:\"https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" The HuggingFace Open LLM leaderboard\"})}),\" uses MMLU to demonstrate how LLMs perform in different areas, including law and computer science. LLMs that score high on this benchmark are skilled at multitasking and reasoning across various domains.\"]})})]}),/*#__PURE__*/t(\"h5\",{children:[\"Toloka is developing \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/toloka-s-advanced-benchmarking-reveals-limitations-and-opportunities-for-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"benchmarks\"})}),\" for specialized domains like natural sciences and university-level math to reveal limitations and opportunities for LLMs.\\xa0\"]}),/*#__PURE__*/e(\"h2\",{children:\"Which are some top-caliber leaderboards?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"The Open LLM Leaderboard\"})}),\",\",/*#__PURE__*/e(a,{href:\"https://lmarena.ai/?leaderboard\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" LMSYS Chatbot Arena\"})}),\", and\",/*#__PURE__*/e(a,{href:\"https://artificialanalysis.ai/leaderboards/models\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" Artificial Analysis LLM Leaderboard\"})}),\" are among the leading LLM leaderboards due to their robust and innovative evaluation methods. These platforms use comprehensive datasets with real-world tasks and a mix of human and automated assessments to provide transparent and reliable rankings. Let's have a look at their evaluation target and methodology:\"]}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"The Open LLM Leaderboard\"})}),\" by Hugging Face is an established resource for insights into LLMs' problem-solving skills and commonsense reasoning. The leaderboard uses six Eleuther AI LM Evaluation Harness benchmarks to test LLMs on math, physics, and general knowledge. It even reveals the environmental impact by reporting data on the C2O emissions from the models' evaluations. This leaderboard is popular among developers as it allows anyone to evaluate their model and see how it competes with existing open-source models.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://lmarena.ai/?leaderboard\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"LMSYS Chatbot Arena\"})}),\" is a model ranking platform with a more hands-on experience. It creates a contest-like environment where the models are ranked based on the popular Elo rating system used in chess. On this crowdsourced platform, users compare responses from two models to the same prompt and vote for the better one. They do this iteratively until a winner is decided. This leaderboard relies on human input to rank models in conversational tasks, reasoning, and context handling.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://artificialanalysis.ai/leaderboards/models\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"The Artificial Analysis LLM Leaderboard\"})}),\" compares over 30 models, including GPT-4o, Gemini, Llama 3, and GPT-3.5 Turbo, on diverse metrics such as price, latency, and context window. The\",/*#__PURE__*/e(a,{href:\"https://artificialanalysis.ai/leaderboards/providers\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" LLM API Providers Leaderboard\"})}),\" compares over 100 LLM endpoints on the same metrics. These platforms help customers and users choose the right AI model and hosting provider for their use cases.\"]})})]}),/*#__PURE__*/t(\"p\",{children:[\"In addition to the above-listed leaderboards, there are several other honorable mentions. For example, the\",/*#__PURE__*/e(a,{href:\"https://tatsu-lab.github.io/alpaca_eval/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" AlpacaEval Leaderboard\"})}),\" assesses the LLMs' ability to follow user instructions. The\",/*#__PURE__*/e(a,{href:\"https://huggingface.co/spaces/mteb/leaderboard\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" MTEB\"})}),\" leaderboard showcases the top text-embedding models based on the Massive Text Embedding Benchmark. As hallucinations remain a weak spot for LLMs, leaderboards like\",/*#__PURE__*/e(a,{href:\"https://huggingface.co/spaces/vectara/leaderboard\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" HHEM\"})}),\" try to measure their resilience against generating false information. LLMs have become prevalent in programming and automation, and practitioners rely on platforms like\",/*#__PURE__*/e(a,{href:\"https://huggingface.co/spaces/mike-ravkine/can-ai-code-results\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" CanAI Code\"})}),\",\",/*#__PURE__*/e(a,{href:\"https://evalplus.github.io/leaderboard.html\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" EvalPlus\"})}),\", and\",/*#__PURE__*/e(a,{href:\"https://gorilla.cs.berkeley.edu/leaderboard.html\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\" Berkley Function-Calling Leaderboard\"})}),\" to find the top-tier programming models in the field.\\xa0\"]}),/*#__PURE__*/e(\"h2\",{children:\"Key takeaways\"}),/*#__PURE__*/e(\"p\",{children:\"Leaderboards are the best way to stay informed about the rapidly evolving field of LLMs. They provide equal ground for evaluating models. A good leaderboard is fair, reliable, and practical. It utilizes standardized benchmarks and metrics that are easy to understand and focus on tasks applicable to real-world scenarios.\"}),/*#__PURE__*/e(\"p\",{children:\"Another essential characteristic is transparency. These platforms clearly explain how models are evaluated and make it easy to reproduce results.\"}),/*#__PURE__*/e(\"p\",{children:\"They also encourage innovation by presenting challenging problems for models to solve. A well-designed leaderboard is inclusive and accessible to participants from eclectic backgrounds. It remains relevant by consistently tracking model progress and updating evaluation methods.\"}),/*#__PURE__*/e(\"h2\",{children:\"Learn more\"}),/*#__PURE__*/t(\"p\",{children:[\"Looking for guidance on LLM evaluation? Our experts are here to help.\\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \\xa0 \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/talk-to-us\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Contact us\"})}),\" to get started.\"]})]});export const richText3=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Envision the exhilaration of a shopper when their product search unlocks the perfect find \u2013 this is the realm of retail where Turkish Hepsiburada aimed for nothing short of excellence. To revolutionize its search capabilities for the millions of products available on its e-commerce platform, Turkish Hepsiburada teamed up with Toloka and developed a solution based on Microsoft Azure. The result was a 9-percent increase in search quality. Hepsiburada is now one of the leading Turkish e-commerce platforms. \"}),/*#__PURE__*/e(\"p\",{children:'In 1998, five visionaries began Hepsiburada in a humble studio flat, aiming to supply the growing demands of the information technology (IT) hardware market. Today, Hepsiburada stands as one of the largest, fastest growing and leading e-commerce platforms in T\\xfcrkiye. Hepsiburada\u2019s platform, which began with a handful of tech products, now boasts over 195 million stock keeping units (SKUs) from more than 101,000 vendors. \"About 12 million active customers access to our vast product range monthly through our application. This application provides not only a marketplace, but also financial services, express delivery, ticketing, and more.\" shares Alexey Shevenkov, Chief Technology Officer at Hepsiburada.'}),/*#__PURE__*/t(\"blockquote\",{children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Imagine yourself searching for a product and instantly finding it. Not just any product, but the right one which is also trusted and popular with other shoppers. That's the seamless experience we aim to offer at Hepsiburada.\"})}),/*#__PURE__*/e(\"p\",{children:\"\u2014 Alexey Shevenkov, CTO, Hepsiburada\"})]}),/*#__PURE__*/e(\"p\",{children:'\"We had an ambitious goal to provide our customers the best online shopping experience ever,\" shares Shevenkov. But the effectiveness of search quality can make or break the retail experience. When a customer looks for a product and it isn\\'t available, that\u2019s disappointing. When the product is available, but not easily found due to subpar search results, that frustrates a customer. So Shevenkov knew that goal wouldn\u2019t be realized without truly understanding the behavior of users on the platform. \"Imagine yourself searching for a product and instantly finding it. Not just any product, but the right one which is also trusted and popular with other shoppers. That\\'s the seamless experience we aim to offer at Hepsiburada,\" he says.'}),/*#__PURE__*/e(\"p\",{children:\"Recognizing that when it comes to navigating the intricate web of e-commerce, two heads \u2014 or in this case, companies \u2014 are better than one, Hepsiburada sought the expertise of Toloka, a technology trailblazer known for its expertise in enhancing machine learning processes with human insights.\"}),/*#__PURE__*/e(\"h2\",{children:\"Revolutionizing retail search together\"}),/*#__PURE__*/e(\"p\",{children:'To leverage the direct relationship between search quality and conversion rates, Toloka helped Hepsiburada design a tool to regularly evaluate search performance and measure a wider range of metrics. \"Relying on metrics like quantity of orders and clicks are limited to only identifying things like popular products. We wanted additional metrics that would enable us to know what customers need and offer that to them even before they click on something,\" explains Shevenkov.'}),/*#__PURE__*/e(\"p\",{children:'To enhance conversions from searches, Toloka proposed to construct search metrics, which would not only train their machine learning models in the future but also routinely assess search performance. \"Our method considers not just standard metrics but also customer preferences, knowledge, requirements, product popularity, and more, to determine the most efficient search system,\" says Olga Megorskaya, CEO at Toloka.'}),/*#__PURE__*/e(\"p\",{children:'Toloka\u2019s platform runs on Microsoft Azure. \"All our labeling, models, demos, and related tasks run through Azure\u2014without it, none of this would be possible,\" shares Megorskaya.'}),/*#__PURE__*/t(\"blockquote\",{children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"All our labeling, models, demos, and related tasks run through Azure \u2014 without it, none of this would be possible.\"})}),/*#__PURE__*/e(\"p\",{children:\"\u2014 Olga Megorskaya, CEO, Toloka\"})]}),/*#__PURE__*/e(\"p\",{children:\"By blending online user behavior data with offline labeling signals, Hepsiburada can discern genuine user intent beyond just clicks. It implemented the solution across their marketplace to support quick shipping, vertical browsing, and other features.\"}),/*#__PURE__*/e(\"h2\",{children:\"Better search results leading to higher conversion rate\"}),/*#__PURE__*/e(\"p\",{children:'The search experience has improved for customers with more accurate results. Hepsiburada meticulously identified and resolved common e-commerce issues like incorrect product sorting and incorrect category predictions, both of which account for 8.3 percent and 15 percent of search failures respectively. \"We\u2019ve achieved a 9-percent increase in search quality. This, in turn, has boosted the search conversion rate,\" Shevenkov proudly shares.'}),/*#__PURE__*/e(\"p\",{children:'Another tool Hepsiburada deployed is competitor comparison. Through crowdsourcing, Hepsiburada and Toloka labeled search results for random queries. \"This allows us to gauge our search quality vis-\\xe0-vis our competitors,\" elaborates Shevenkov. \"We\\'ve already observed an 8 percent improvement in measured relevance, which correlates with increase in sales and customer satisfaction.\"'}),/*#__PURE__*/e(\"p\",{children:'As the journey from search to purchase becomes more seamless, Hepsiburada keeps thinking of ways to enhance the customer experience. \"In the realm of e-commerce, metrics play a pivotal role. By utilizing the right metrics and collaborating with the right partners, retailers can embark on a cycle of constant improvement. Then, for the shopper, we deliver an even better, seamless shopping journey,\" concludes Shevenkov.'}),/*#__PURE__*/t(\"blockquote\",{children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"We\u2019ve achieved a 9-percent increase in search quality. This, in turn, has boosted the search conversion rate.\"})}),/*#__PURE__*/e(\"p\",{children:\"\u2014 Alexey Shevenkov, CTO, Hepsiburada\"})]}),/*#__PURE__*/t(\"p\",{children:[\"The content of this article was originally posted on \",/*#__PURE__*/e(a,{href:\"https://customers.microsoft.com/en-us/story/1688738181933208375-hepsiburada-retailers-azure-en-turkiye\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Microsoft Customer Stories\"})}),\".\"]})]});export const richText4=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Imagine a world where artificial intelligence isn't confined to a single mode of interaction but can seamlessly bridge the gap between text and images, language and vision, and even sound and sensation. AI is no longer limited to just processing text or images but can understand, generate, and interpret information across multiple domains. That\u2019s where the concept of foundation models steps in.\"}),/*#__PURE__*/t(\"p\",{children:[\"These models, often associated with \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/large-language-models/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"large language models\"})}),\", have evolved to encompass a wide array of applications, proving their versatility beyond just text processing. In this article, we will explore the evolving landscape of foundation models, their diverse capabilities, and the impact they have on the AI industry by unlocking the power of multimodality.\"]}),/*#__PURE__*/e(\"h2\",{children:\"What is a Foundation Model\"}),/*#__PURE__*/t(\"p\",{children:[\"A foundation model is a large, pre-trained neural network designed to understand and generate various types of content. These models are trained on massive datasets, which allows them to capture the intricacies and nuances of data. A group of data scientists at Stanford Institute for Human-Centered Artificial Intelligence (HAI) Center for Research on Foundation Models came up with the term in research on foundation models called \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/abs/2108.07258\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"On the Opportunities and Risks of Foundation Models\"})}),'. They referred to it as \u201Cany model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks\".']}),/*#__PURE__*/e(\"h3\",{children:\"Pretraining and Fine-Tuning of Foundation Models\"}),/*#__PURE__*/t(\"p\",{children:[\"As indicated in the definition above, foundation models are primarily based on \",/*#__PURE__*/e(\"strong\",{children:\"self-supervised learning (SSL)\"}),\". It is a powerful approach to AI systems training that allows data scientists to leverage huge data sets reducing or eliminating time-consuming labeling. During SSL model learns to generate its own labels from the input data, eliminating the need for external labels.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Foundation models are usually \",/*#__PURE__*/e(\"strong\",{children:\"pre-trained\"}),\" on large, diverse datasets, such as texts taken from various websites. Pretraining is a machine learning technique in which a model is trained on a large dataset to learn general features of training data before being fine-tuned for a specific task.\"]}),/*#__PURE__*/e(\"p\",{children:\"Pre-training begins with the collection of a vast and diverse dataset that is relevant to the domain the model is intended for. For text-based foundation models, this dataset often includes extensive text data from sources like books, articles, websites, and other textual sources. For multimodal models, the dataset may consist of text, images, audio, and other types of data.\"}),/*#__PURE__*/e(\"p\",{children:\"In self-supervised learning, the primary idea is to train a model using unlabeled data and generate tasks similar to the ones presented in supervised learning from that unlabeled data without relying on external labels. However, there may be situations where labeled data is used in conjunction with self-supervised learning, but the labeled data is typically introduced during the fine-tuning phase or for specific downstream tasks.\"}),/*#__PURE__*/t(\"p\",{children:[\"Pretrained model can be \",/*#__PURE__*/e(\"strong\",{children:\"fine-tuned\"}),\" with the right data to make them serve their intended purpose. Such foundation models may then serve as the basis or foundation for many AI applications, hence their name. This fine-tuning step involves adjusting the model's parameters to perform well on the task at hand. The general knowledge and features learned during pre training are adapted for the specific task that is typically supervised learning, because labeled data is introduced during that stage.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Role of Deep Learning in Foundation Models\"}),/*#__PURE__*/t(\"p\",{children:[\"The concept of the foundation model is also based on \",/*#__PURE__*/e(\"strong\",{children:\"deep learning\"}),\". It is a subfield of machine learning that leverages neural networks with multiple layers to process and learn from vast datasets. Deep learning models can scale to handle vast amounts of data. This scalability is essential for foundation models, allowing them to process and learn from large and diverse datasets.\"]}),/*#__PURE__*/e(\"p\",{children:\"This architecture is the driving force behind foundation models and enables them to excel in understanding and generating information. These models use deep learning techniques to learn complex patterns, hierarchies, and representations of data, making them adaptable to diverse tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"Deep learning provides the flexibility and adaptability to integrate different modalities into a single model. Foundation models therefore not only can be applied to a range of natural language processing (NLP) tasks but also can handle multiple modalities, such as text and images or text, images, video, and audio.\"}),/*#__PURE__*/e(\"p\",{children:\"A deep neural network consists of multiple layers of interconnected neurons that can be fine-tuned to process different types of data, including text, images, audio, and more. This adaptability allows deep foundation models to be versatile enough to handle multimodal input. Their multimodal capability is a key feature that distinguishes them from specialized models designed for single tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"The thing is, traditional machine learning models, while capable of solving specific tasks, often lack the massive scale and capacity of foundation models. They are designed for specific tasks and may not be able to apply well to diverse applications.\"}),/*#__PURE__*/e(\"p\",{children:\"To summarize all of the above, the Foundation models represent a particular type of multimodal AI model that combines pre-training through self-supervision, deep learning architecture and the ability to be fine-tuned later. Such models have started to rapidly evolve in the last decade. Therefore, specialists at the Stanford Center for Research on Foundation Models decided to give this new promising tendency a name - foundation models. Let's take a closer look at other prominent features of foundation models that can help us distinguish them from all other types of AI models.\"}),/*#__PURE__*/e(\"h2\",{children:\"Distinguishing Characteristics of Foundation Models\"}),/*#__PURE__*/e(\"p\",{children:\"In addition to the characteristics above mentioned, foundation models possess several distinctive features that set them apart from other types of AI models. These characteristics make them versatile and adaptable, allowing them to serve as a foundation upon which various specialized AI applications can be built. Here are the features that distinguish foundation models from other AI models.\"}),/*#__PURE__*/e(\"h3\",{children:\"Transfer Learning\"}),/*#__PURE__*/e(\"p\",{children:\"Foundation models are designed with the idea of transfer learning. It is a machine learning technique where a model trained on one task or dataset is adapted to perform another related task. The idea behind transfer learning is to leverage the knowledge gained during the initial training on one task to improve performance on a new, but related, task.\"}),/*#__PURE__*/e(\"p\",{children:\"They are pre-trained on a large corpus of text or data, making them adaptable to various tasks and domains. This transfer learning approach reduces the need for training models from scratch for specific applications.\"}),/*#__PURE__*/e(\"p\",{children:\"After pre-training, foundation models can be fine-tuned for certain tasks, thanks to the concept of transfer learning. This adaptability sets them apart from models that are solely designed for one specific purpose. Fine-tuning allows foundation models to excel in a wide range of tasks, such as language translation, image classification, and more.\"}),/*#__PURE__*/e(\"h3\",{children:\"Transformer Architecture\"}),/*#__PURE__*/e(\"p\",{children:'Foundation models are typically (not mandatorily) based on transformer architecture, which now has become the standard for many natural language processing and multimodal models. Transformers are a type of neural network architecture. The introduction of the Transformer architecture in the paper \"Attention Is All You Need\" in 2017 marked a significant breakthrough in NLP.'}),/*#__PURE__*/t(\"p\",{children:[\"Transformers relied on self-attention mechanisms that allow them to capture long-range dependencies and relationships within the data, leading to improved language understanding. They set foundation models apart from other neural network architectures that may not handle sequences and relationships in the same manner. The transformer architecture laid the foundation for modern \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/large-language-models/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"large language models\"})}),\" (LLMs).\"]}),/*#__PURE__*/e(\"p\",{children:\"The application of Transformers to modalities other than NLP has been made possible by the development of a Transformer variety called the Vision Transformer (ViT). The ViT model has revolutionized the field of computer vision by demonstrating the efficacy of the Transformer architecture in understanding and processing visual data. ViT's success has inspired the development of various ViT-based models, and it continues to be a subject of active research and innovation in the field of AI and machine learning.\"}),/*#__PURE__*/e(\"h2\",{children:\"Benefits of Foundation Models\"}),/*#__PURE__*/e(\"p\",{children:\"The unique characteristics of foundation models unlock a wide array of benefits that have a profound impact on various AI applications. These benefits are a direct result of their versatility, adaptability, and capabilities in understanding and generating diverse data types.\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Customization for Diverse Applications\"}),\". The adaptability of foundation models through fine-tuning allows for customization to specific tasks or domains. This flexibility ensures that AI systems are not one-size-fits-all but can be tailored to diverse industries and use cases;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Versatility\"}),\". Foundation models are highly versatile and can be applied to a broad spectrum of tasks due to their multimodality. They can perform tasks related to natural language understanding and generation, image processing, and even multimodal tasks that combine text, image, and audio data;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Efficient Transfer Learning\"}),\". The pre-training and fine-tuning approach in foundation models significantly reduces the time and resources required to develop AI solutions. This efficiency is especially valuable for businesses and developers looking to implement AI in their applications;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cost and Time Savings\"}),\". Foundation models significantly reduce the time required to develop AI solutions. Instead of building models from scratch, developers can leverage pre-trained models, saving weeks or even months in development time. Once deployed, models based on foundation models often require less ongoing maintenance. The models are designed to be robust and adaptable, reducing the need for continuous adjustments;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Minimal Data Annotation\"}),\". Traditional machine learning models often demand extensive labeled data for training. In contrast, foundation models require fewer labeled examples for fine-tuning, thanks to their knowledge gained during pre-training. This is especially advantageous when labeled data is scarce or costly to obtain.\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"Applications of Foundation Models\"}),/*#__PURE__*/e(\"h3\",{children:\"Natural Language Processing\"}),/*#__PURE__*/e(\"p\",{children:\"Natural Language Processing applications are one of the most prominent and rapidly evolving domains for foundation models. These models, equipped with advanced language understanding capabilities, are employed in a wide range of NLP applications. Here are some of them:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Machine Translation. Foundation Models like BERT have improved the quality of machine translation applications;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Language Generation. They are employed in chatbots and automated content generation;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Text Summarization. They can generate concise summaries of long text documents;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Sentiment Analysis. Foundation models excel in sentiment analysis, determining the emotional tone in text data;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Speech Recognition. In speech recognition, foundation models convert spoken language into text, serving as the backbone for voice assistants, transcription services, and voice command systems.\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Computer vision\"}),/*#__PURE__*/e(\"p\",{children:\"Computer vision (CV) is a field of artificial intelligence and computer science that focuses on enabling computers and machines to interpret and understand visual information from the world, similar to how humans perceive and process images and videos. Here are some key CV applications of foundation models:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Image Classification. Foundation models like CLIP are capable of classifying images and associating them with their corresponding text descriptions;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Object Detection. They can identify and locate objects within images;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Image Captioning. Foundation models generate descriptive captions for images, making it easier to understand the content of pictures;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Image Generation. Models like DALL-E can create images based on textual prompts;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Face Recognition. CV foundation models are used in face recognition and facial expression analysis systems;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Semantic Segmentation. Foundation models can perform semantic segmentation by labeling each pixel in an image with the object or class it belongs to;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Video Description. Foundation models can be employed for generating text descriptions for video content;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Visual Question Answering (VQA). VQA systems powered by foundation models can answer questions about the content of images.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Some of the applications can be considered multimodal because the input and/or output can fall into different categories of modality. Multimodal applications of foundation models leverage their versatility to understand and make connections between different types of data. For example, tasks like image generation from text prompts or image captioning imply the use of visual and textual modality.\"}),/*#__PURE__*/e(\"h2\",{children:\"Examples of Foundation Models\"}),/*#__PURE__*/e(\"h3\",{children:\"Large Language Models (LLMs)\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://toloka.ai/large-language-models/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Large Language Model\"})}),\" is a category of foundation models that has gained significant attention in the field of natural language processing. LLMs are incredibly large in terms of the number of parameters, often ranging from hundreds of millions to tens of billions. This immense scale allows them to capture intricate patterns in language data.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"GPT (Generative Pre-trained Transformer)\"})}),/*#__PURE__*/e(\"p\",{children:\"Developed by OpenAI, GPT is one of the most famous foundation models. There are 5 basic versions of GPT, each representing a significant leap in scale and performance in comparison to the previous one. GPT-3.5 was used to create the chatbot product known as ChatGPT. It became popular for its ability to engage in conversational interactions and provide informative responses to a variety of user queries. GPT-4, the latest version of GPT, is a multimodal foundation model, capable of text and image processing.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"BERT (Bidirectional Encoder Representations from Transformers)\"})}),/*#__PURE__*/e(\"p\",{children:\"BERT, developed by Google, focuses on bidirectional understanding of text. Models like BERT have a profound impact on our daily lives, often behind the scenes. It's widely used in search engine algorithms to improve the accuracy of search results. BERT analyzes and translates text in machine translation systems like Google Translate, considering context and grammar. This foundation model is a fundamental tool for language understanding in various contexts, including chatbots and virtual assistants. It helps these systems comprehend user queries and generate contextually relevant responses. In applications like Gmail, BERT can predict and suggest text as users compose emails, making text input more efficient and user-friendly. While many foundation models are primarily associated with NLP and understanding, as was mentioned earlier some models have been developed to work with images and other types of data. These multimodal foundation models are also typically based on deep learning architectures, similar to their text-only processing counterparts.\"}),/*#__PURE__*/e(\"h3\",{children:\"Vision-Language Foundation Models\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"CLIP (Contrastive Language-Image Pretraining)\"})}),/*#__PURE__*/e(\"p\",{children:\"Foundation models like CLIP, introduced by OpenAI in 2021, extended the Transformer architecture to understand and generate text and images. This opened up new possibilities for AI in CV, multimodal applications, and cross-modal understanding. CLIP, like other foundation models in a vision-language domain, is pre-trained on a large corpus of text and image data, which enables it to learn the relationships between visual and textual content.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"DALL\\xb7E\"})}),/*#__PURE__*/e(\"p\",{children:\"Also developed by OpenAI, DALL\\xb7E is a model capable of generating images from textual descriptions. It is a multimodal implementation of GPT. It takes text prompts and produces corresponding images. It can produce highly creative and contextually relevant images based on textual descriptions.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Stable Diffusion\"})}),/*#__PURE__*/e(\"p\",{children:\"Stable Diffusion is a foundation model with generative AI capabilities. This text-to-image tool employs a kind of deep generative AI neural network called the latent diffusion model. Users can instruct the model to create diverse high-quality images. The model allows the use of prompts to partially alter existing images via inpainting (adding new elements to an image) and outpainting (removing elements from an image).\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"GPT-4\"}),' One notable advancement in GPT-4, distinct from its predecessors, is its potential to handle image inputs. It\\'s important to note that this feature is currently labeled as a \"research preview\" and is not yet accessible to the general public. GPT-4 impressively demonstrates its proficiency in accurately understanding intricate visuals, including charts, memes, and scientific papers.']}),/*#__PURE__*/e(\"h3\",{children:\"Vision-Audio-Language Foundation Models\"}),/*#__PURE__*/e(\"p\",{children:\"Vision-Audio-Language (VAL) foundation models are a subclass of multimodal models that have the ability to understand and work with data from three primary modalities: images, audio, and text. These models are designed to process, generate, and make connections between data in these modalities\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"VALOR (Vision-Audio-Language Omni-Perception Pretraining Model)\"})}),/*#__PURE__*/e(\"p\",{children:\"VALOR is designed to understand and create information from multiple types of data, such as pictures (vision), sounds (audio), and words (language). It's different from previous models that mainly focus on connecting pictures and words. VALOR combines all three types of data to get a better understanding.\"}),/*#__PURE__*/e(\"p\",{children:\"VALOR is made up of three separate parts that work with each type of data. These parts are called encoders. There is also a part called a decoder that can put all this information together and create text based on what the model sees, hears, or both.\"}),/*#__PURE__*/e(\"h2\",{children:\"Challenges of Foundation Models\"}),/*#__PURE__*/e(\"p\",{children:\"Foundation models, while powerful and versatile, come with several challenges that need to be addressed for their effective and responsible use.\"}),/*#__PURE__*/e(\"h3\",{children:\"Fake Content\"}),/*#__PURE__*/e(\"p\",{children:\"Misinformation is a significant challenge associated with the use of foundation models, particularly those designed for natural language generation. Foundation models can inadvertently amplify misinformation by generating text that sounds authoritative and convincing. These models also enable the creation of deepfake content, including manipulated videos, audio, and images. Deepfakes can be used to impersonate public figures or create false evidence, leading to misinformation and distrust. Unfortunately, traditional methods for detecting misinformation are less effective against content generated by advanced foundation models, as they can closely mimic humans.\"}),/*#__PURE__*/e(\"h3\",{children:\"Legal and ethical considerations\"}),/*#__PURE__*/e(\"p\",{children:\"Generating content using foundation models can raise questions about intellectual property rights. If a model generates text, art, or music, who owns the resulting content, and are there copyright implications? Ownership and control over the content generated by foundation models are unclear. This becomes a legal issue when content is used for commercial purposes or when disputes arise over authorship.\"}),/*#__PURE__*/e(\"p\",{children:\"Discriminatory or biased content generated by foundation models can lead to legal challenges, especially in the context of anti-discrimination laws. When foundation models are used to generate harmful, illegal, or fraudulent content, legal issues arise. The responsibility for preventing misuse and taking legal action against malicious users can be complex.\"}),/*#__PURE__*/e(\"h3\",{children:\"Data Privacy\"}),/*#__PURE__*/e(\"p\",{children:\"The massive scale of foundation models raises concerns about data privacy. Pretraining on large datasets means that sensitive information may be present in the model's parameters, potentially posing privacy risks. Obtaining informed consent from individuals for collecting and using their data can be challenging. Users may not fully understand the extent to which their data will be used, making it difficult to obtain meaningful consent.\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/e(\"p\",{children:\"Foundation models are an outstanding breakthrough in the development of artificial intelligence, significantly transforming the landscape of natural language processing, computer vision, and multimodal tasks. These models, often based on the powerful Transformer architecture, have the capacity to understand, generate, and interpret a wide range of data.\"}),/*#__PURE__*/e(\"p\",{children:\"Their applications are vast, and their benefits are substantial, though they come with their share of challenges. As these models continue to evolve, it's crucial to use them responsibly and ethically to utilize their full potential for the betterment of society. Troubleshooting challenges associated with disinformation and fake content, ethical considerations, data privacy concerns, and regulatory compliance remains a critical issue. Accountability for the development and implementation of these models is vital to ensure that their impact on society is positive and fair. Foundation models have given us the ability to interact with machines more naturally, creating new possibilities for content generation, and pushing the boundaries of human knowledge. They serve as a reminder of the infinite range of opportunities that AI brings to the table.\"})]});export const richText5=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Generative artificial intelligence emerged quite recently and has already revolutionized various business industries. It is in a very early stage of its development, but it's already not just a buzzword; it's a powerful tool that is redefining how businesses operate, strategize, and engage with customers.\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI has already demonstrated its potential to enhance efficiency and creativity across multiple domains. It can create images and photos, synthesize music and voice, but an incredible breakthrough happened when generative AI tools gained the ability to comprehend and generate cohesive and human-like texts.\"}),/*#__PURE__*/e(\"p\",{children:\"Its unparalleled ability to create textual content with context, coherence, and creativity has made it indispensable in modern business operations. As we explore generative AI applications, we'll focus on how it helps businesses stay competitive, agile, and ready for the future, particularly as it pertains to text generation.\"}),/*#__PURE__*/e(\"h2\",{children:\"Generative AI in a Nutshell\"}),/*#__PURE__*/e(\"p\",{children:\"Generative Artificial Intelligence, as the name suggests, generates content. It is opposed to traditional AI, which does not create any new information but accomplishes important tasks of systematization, recognition, and prediction.\"}),/*#__PURE__*/e(\"p\",{children:\"The main difference between the two types of AI is that the former uses generative models to accomplish its job, while the latter employs discriminative models. Discriminative Models focus on learning the boundary that separates different classes or categories in data. Generative models on the other hand aim to learn the underlying probability distribution of the data. They can generate new data samples that are similar to the training data.\"}),/*#__PURE__*/e(\"p\",{children:\"Another pivotal part of the generative AI ecosystem is foundation models. Generative AI utilizes their power to generate content, whether it's generating creative stories, composing music, or realistic photos or human-like responses in conversational agents.\"}),/*#__PURE__*/e(\"p\",{children:\"The adaptability of generative AI foundation models allows them to be fine-tuned for specific generative tasks. They serve as a base for other applications. Apps built on top of them may have absolutely zero in common, yet have the same model at the core. The vast knowledge and linguistic capabilities embedded in such models enable generative AI to produce coherent and contextually relevant content, making it a valuable tool for various applications.\"}),/*#__PURE__*/e(\"p\",{children:\"Multimodal foundation models are designed to process and understand data from multiple modalities, which can include not only text but also images, audio, and more. These models have the capacity to bridge the gap between different types of data and enable comprehensive analysis and generation across various media types.\"}),/*#__PURE__*/e(\"h2\",{children:\"Examples of Generative AI models\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"GPT-4 (Generative Pre-trained Transformer 4)\"}),\". Developed by OpenAI, GPT-4 is one of the largest multimodal models with 1.76 trillion parameters. This latest version of GPT is capable of processing requests in the form of images and text, and then producing new text responses;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"BERT (Bidirectional Encoder Representations from Transformers)\"}),\". Google's BERT is a large language model that revolutionized natural language understanding by pre-training on vast amounts of text data. It's used for various NLP tasks, including sentiment analysis, question-answering, and text classification;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"PaLM-E (Pathways Language Model - Embodied)\"}),\". PaLM-E is a multimodal language model for controlling robots. It is a combination of Google's PaLM and Vision Transformer (ViT) models. Input data for the PaLM-E model are multimodal sentences that combine text and visual embeddings, which allows the model to successfully solve various tasks like planning sequential manipulations, visual responses to questions, etc.\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"Generative AI Use Cases\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI capabilities encompass a broad spectrum of advanced functions that employ the power of artificial intelligence to create, generate, and enhance content across various domains. Here are some of its key capabilities and use cases:\"}),/*#__PURE__*/e(\"h3\",{children:\"Text generation\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI can craft human-like text, making it invaluable for content generation, conversational AI tools like chatbots, and creative writing. The natural language capabilities of generative AI allow us to create content that is not only contextually accurate but also creatively compelling. Natural language processing (NLP), a subset of AI, provides machines with the ability to understand and manipulate human language. In the context of generative AI, this entails text generation that possesses an astonishingly human-like fluency and coherence.\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI can help businesses extract valuable insights from large volumes of text data, translate content from one language to another quickly and accurately, streamline the process of preparing emails, presentations, and documents and more.\"}),/*#__PURE__*/e(\"h3\",{children:\"Music Generation\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI can compose original melodies, harmonies, and musical compositions. It's not limited to a specific genre and can generate music in classical, jazz, pop, or any style you desire. Musicians and producers can collaborate with generative AI tools to expedite music production. Generative AI can come up with an almost endless stream of musical ideas, making it a valuable tool for composers seeking inspiration or musicians looking to collaborate with an AI co-creator.\"}),/*#__PURE__*/e(\"h3\",{children:\"Realistic speech audio generation\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI powers Text-to-Speech (TTS) systems that convert written text into spoken words. These systems produce speech that closely mimics human vocal characteristics, including tone, pitch, and rhythm. Advanced TTS models have a nuanced understanding of language, allowing them to generate speech with natural pronunciation and intonation. Generative AI technology enables voice customization, where users can modify speech characteristics, such as gender, accent, and age, to align with specific needs or brand identities.\"}),/*#__PURE__*/e(\"h3\",{children:\"Code generation\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI can automate repetitive tasks, such as writing boilerplate code, generating database queries, or formatting code according to specific coding standards. Integrated code generation tools can provide real-time code suggestions and auto-completion as developers write code. This enhances productivity and streamlines software development. Generative AI can also identify and fix code defects or suggest improvements for code optimization.\"}),/*#__PURE__*/e(\"h3\",{children:\"Computer Vision and Image Generation\"}),/*#__PURE__*/e(\"p\",{children:\"Computer vision is a field of AI that focuses on enabling computers to interpret and understand visual information from the world, much like the human visual system. It involves the development of algorithms and models to enable computers to analyze, process, and make decisions based on visual data from images and videos.\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI models can produce realistic images and visually striking artworks, paintings, and illustrations. All this owing to the discovery and development of algorithms such as Generative Adversarial Networks (GANs) and a subcategory of deep generative models called Diffusion Models.\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI assists in generating visuals for marketing campaigns, advertisements, and promotional materials. It can create eye-catching images and graphics tailored to specific brand messages and target audiences.\"}),/*#__PURE__*/e(\"p\",{children:\"In computer vision tasks like object detection and image classification, generative AI can create synthetic images. These images are used to expand training datasets, improving the performance and robustness of machine learning models. Generative AI generates realistic content for virtual and augmented reality environments. It creates 3D models, textures, and simulations that make virtual experiences more immersive.\"}),/*#__PURE__*/e(\"p\",{children:\"In e-commerce and image-based search engines, computer vision identifies products or objects within images. Generative AI enhances recommendation systems by providing more accurate and relevant suggestions based on such visual content.\"}),/*#__PURE__*/e(\"h2\",{children:\"Why does Text Generation Play a Key Part in Generative AI Development?\"}),/*#__PURE__*/e(\"p\",{children:\"All cases of human-like text usage and processing by a computer are called Natural Language Processing (NLP). In the 1980s, machine learning was introduced for computer language processing.\"}),/*#__PURE__*/e(\"p\",{children:\"By the 2020s generative AI has advanced rapidly, as the capacity and computational power of devices has increased due to new technological developments. The field of NLP has received a significant boost. Generative AI tools capable of generating texts that are indistinguishable from those written by humans have appeared. They can now perform very high-quality translations from one language to another, extract data from the text presented to them, create document templates, etc. So why is generative AI especially focused on text processing?\"}),/*#__PURE__*/e(\"p\",{children:\"The answer may lie in the fact that text is the basis of perhaps all human thought processes. Everything begins with text and language: a painting first appears as a thought in the artist's head, melodies are also recorded using the language of notes, and even our inner voice is also a text. This is probably why one of the first breakthroughs in the development of generative AI was the ability of AI tools to generate and process text in every possible way.\"}),/*#__PURE__*/e(\"p\",{children:\"Today, in order to generate a picture in a generative AI application, we have to enter a prompt containing a textual message that tells the algorithm what we want to see in the picture. Generative AI also generates music and sounds based on text requests. This is quite similar to how a person's thought process works.\"}),/*#__PURE__*/e(\"p\",{children:\"Artificial intelligence aims to learn to think and create like a human. Therefore, figuratively speaking, in order to achieve this goal in the future, data scientists have decided to start (or should we rather say continue) the development of generative AI by mastering its use of text, as humans also start any kind of interaction with the world around them using text.\"}),/*#__PURE__*/e(\"p\",{children:\"Almost all modern generative AI tools can work with text, but not all can work with visual data. Although multimodal systems capable of processing both text and image are gradually gaining momentum.\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI can accomplish certain tasks better and faster than a human being. It may become a competitive advantage for many businesses. Nowadays, generative AI is becoming an integral part of both large and small businesses.\"}),/*#__PURE__*/e(\"h2\",{children:\"Business Applications of Text Generation\"}),/*#__PURE__*/e(\"p\",{children:\"By integrating text generation with generative AI into various aspects of business operations, organizations can improve productivity, reduce costs, provide better customer experiences, and gain a competitive edge in today's data-driven and digitally connected world. That's how organizations can employ generative AI in their processes:\"}),/*#__PURE__*/e(\"h3\",{children:\"Generating Documentation\"}),/*#__PURE__*/e(\"p\",{children:\"Generating documentation through generative AI tools can streamline the process of creating manuals, guides, reports, and other types of documents. Generative AI reduces manual effort, minimizes errors, and ensures that critical information is always available to employees, customers, and stakeholders.\"}),/*#__PURE__*/e(\"p\",{children:\"Leveraging generative AI document generation can provide a competitive edge by improving efficiency and the ability to adapt to changing market dynamics. These are some insights concerning documentation generation with the help of generative AI:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Product descriptions\"}),\". E-commerce businesses can employ AI to generate product descriptions and catalogs. This is particularly useful for maintaining large product inventories and ensuring consistency in product offerings;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Reports and presentations\"}),\". Generative AI assists in generating reports, market analyses, financial summaries, and business plans. It can summarize complex data into concise, readable reports for stakeholders and decision-makers;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Email sequences\"}),\". AI-powered systems can assist employees in composing professional emails, offering suggestions for subject lines, body content, and titles;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Legal and finance\"}),\". These functions benefit from generative AI in drafting contracts, agreements, and reports;\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Documents versioning\"}),\". AI can automatically update documents in real time based on changing data or variables, ensuring that documents are always current and accurate.\"]})})]}),/*#__PURE__*/e(\"h3\",{children:\"Translating Documents\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI can translate documents and text from one language to another accurately and quickly, facilitating global communication and expanding market reach, allowing businesses to reach a broader audience. Traditional human translation can be time-consuming, especially for lengthy documents. Generative AI can translate documents rapidly, saving valuable time and resources.\"}),/*#__PURE__*/e(\"p\",{children:\"AI-powered document translation allows businesses to tap into new markets while maintaining language accuracy and consistency. Some use cases of translating documents with the help of generative AI for business include:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Translating financial reports, investment summaries, and market analysis documents allows businesses to communicate financial insights to international stakeholders effectively;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Businesses can translate marketing materials, advertisements, product descriptions, and website content to reach a wider global audience. AI ensures that the translated content aligns with the brand's messaging and style.\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Marketing Strategies and Content Creation\"}),/*#__PURE__*/e(\"p\",{children:\"AI-powered tools generate large volumes of marketing content quickly, including landing pages, blog and social media posts, and ad copies, helping businesses maintain a strong online presence and engage with their target audience effectively. Here are some features of generative AI regarding generating textual content for marketing purposes:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Generative AI can produce a wide range of content at a scale that would be challenging for human writers alone. This enables businesses to consistently update their website and engage with their audience through content marketing;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"AI-powered brainstorming sessions can assist marketing teams in generating a wide range of ideas quickly. AI can suggest various angles, themes, and approaches for marketing campaigns;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Generative AI can assist in creating content plans by generating ideas and organizing them into a structured timeline for consistent and strategic content delivery.\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Autonomous Agents and Chatbots\"}),/*#__PURE__*/e(\"p\",{children:\"Chatbots and virtual assistants utilize generative AI to engage users in natural conversations. They provide customer support, answer queries, and offer personalized recommendations, enhancing customer service. Incorporating autonomous agents and chatbots into business processes enhances efficiency, scalability, and customer satisfaction. Such AI-powered agents can handle routine tasks, provide instant responses, and generate content that resonates with customers, ultimately driving business growth and competitiveness. Their features include:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Autonomous agents and chatbots can provide 24/7 customer support, ensuring that customers receive assistance even outside business hours;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Chatbots can answer common customer queries by generating responses based on a pre-defined FAQ database, reducing the workload on customer support teams;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Virtual assistants can troubleshoot and resolve customer issues by providing step-by-step instructions or generating solutions.\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Personalized Product Recommendations\"}),/*#__PURE__*/e(\"p\",{children:\"E-commerce platforms use AI-generated product recommendations based on user browsing and purchasing history, which allows them to boost customer satisfaction. Personalized product recommendations powered by generative AI models offer businesses a powerful tool to enhance customer engagement, increase sales, and improve user experience. Customers are more likely to return to a business that consistently recommends products they find relevant and valuable, stimulating long-term loyalty.\"}),/*#__PURE__*/e(\"p\",{children:\"Therefore personalized recommendations created with the help of generative AI can significantly contribute to increasing the average purchase value and click-through rates (CTR) in an online store as customers are more likely to engage with and act on personalized recommendations that align with their interests and needs. Here's what AI can do in terms of personalized recommendations:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Generative AI creates user profiles by analyzing the data. These profiles capture individual preferences, interests, and behaviors. The AI identifies patterns and trends in the data to understand what each user is likely to be interested in;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"AI can divide customers into groups based on behavior, demographics, and preferences, allowing businesses to provide targeted recommendations to each such group;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Generative AI can provide real-time recommendations as users browse an online store or platform. This guarantees that customers are presented with the most relevant products based on their current interests and browsing behavior.\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Market Research and Sentiment Analysis\"}),/*#__PURE__*/e(\"p\",{children:\"AI-driven text analysis tools can process and summarize vast amounts of textual data, extracting valuable insights about market trends, customer sentiment, and competitor strategies. AI can identify emerging trends, consumer preferences, and market shifts by analyzing textual data. This information helps businesses stay ahead of the competition and adapt their strategies accordingly. The following generative AI use cases aid in conducting effective market research and sentiment analysis:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Generative AI can analyze customer feedback, reviews, and social media conversations to determine sentiment. It identifies positive, negative, or neutral sentiments and helps businesses understand how customers perceive their products or services;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis carried out by generative AI can pinpoint areas where products or services excel and areas that need improvement. This information is invaluable for refining offerings to meet customer expectations;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Businesses can use generative AI text processing tools to monitor competitors' activities and sentiments expressed about them. This data aids businesses in refining their competitive strategies.\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Content Moderation\"}),/*#__PURE__*/e(\"p\",{children:\"Content moderation with generative AI is essential for businesses to maintain a safe and respectful online environment, comply with regulations, and protect their brand reputation. Businesses often have social media pages for engagement and marketing. AI text processing assists in automating content moderation on such platforms.\"}),/*#__PURE__*/e(\"p\",{children:\"It can detect and filter out inappropriate or harmful content, offensive, or spam comments, ensuring a positive online presence and appropriate social media posts. By preventing the spread of harmful or offensive content associated with the brand, content moderation safeguards brand reputation and customer trust. Some prominent generative AI use cases for content moderation include:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"E-commerce businesses can use content moderation to ensure that customer reviews are genuine and constructive while filtering out fake reviews or hate speech;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"AI can assist in filtering out spam, phishing emails, and other malicious content from business email inboxes, enhancing email security;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Video platforms and live streaming services can use AI to detect and remove inappropriate comments, hate speech, and harmful content from chat sections during live broadcasts\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"Advantages of Generative AI Tools in Business\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI offers numerous advantages to businesses and various industries. These tools, powered by natural language processing, have the potential to revolutionize operations, enhance customer experiences, and drive growth in organizations. Here are some key advantages of using generative AI tools in business:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Scalability\"}),\". Generative AI can quickly generate content at scale, allowing businesses to handle increased workloads and adapt to changing demands. Businesses can also easily scale content generation and customer support efforts without proportionally increasing human labor. AI can handle increased workloads effortlessly;\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cost Efficiency\"}),\". Generative AI can automate content generation, reducing the need for manual content creation, writing, and editing. This leads to cost savings in content production and marketing efforts;\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Time Savings\"}),\". AI-driven automation accelerates various tasks, such as drafting documents, generating reports, and responding to customer inquiries, freeing up employees' time for more strategic and value-added activities;\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Improved Efficiency\"}),\". Generative AI automates time-consuming and repetitive tasks, allowing employees to focus on higher-value activities and strategic initiatives;\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Competitive Advantage\"}),\". Businesses adopting generative AI techniques can gain a competitive advantage by being able to stay on top of technological advancements and offer a superior customer experience due to its use.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Standout Generative AI Use Cases\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI solutions have proven to be transformative tools for big companies, as they continue to discover unique and unconventional applications across various industries.\"}),/*#__PURE__*/e(\"h3\",{children:\"E-commerce\"}),/*#__PURE__*/e(\"p\",{children:'The e-commerce megacorporation Amazon has introduced a feature in its shopping app that employs AI to condense customer reviews into concise summaries. These summaries offer a quick overview of what customers appreciated and disliked about a product, accompanied by a disclaimer indicating that the summary has been \"AI-generated from the text of customer reviews.\"'}),/*#__PURE__*/e(\"p\",{children:\"The use of AI to summarize reviews is a valuable tool for online shoppers, considering Amazon's vast catalog of products, each potentially having thousands of reviews. It simplifies the decision-making process for customers. It's also worth mentioning that Amazon has a long history of using AI and machine learning to provide its customers with personalized recommendations and targeted advertising.\"}),/*#__PURE__*/e(\"h3\",{children:\"Healthcare\"}),/*#__PURE__*/e(\"p\",{children:\"Pharmaceutical giants like Pfizer employ a generative AI model to manufacture medicines that are more likely to perform well. The company can assess a vast amount of clinical data to uncover deeper insights and enhance the efficiency of the discovered drugs. That way generative AI accelerates the process of drug selection process and Pfizer identifies potential drug candidates with a higher probability of success.\"}),/*#__PURE__*/e(\"p\",{children:\"Pfizer also employs generative AI models to identify and screen for rare diseases, such as transthyretin amyloid cardiomyopathy (ATTR-CM). The implementation of these technologies allows medical professionals and patients to detect the symptoms of rare diseases more accurately and at an earlier stage.\"}),/*#__PURE__*/e(\"h3\",{children:\"Film industry\"}),/*#__PURE__*/e(\"p\",{children:\"In content creation, film studios like Warner Bros. have embraced generative AI to forecast the box office of their movies. Machine learning algorithms analyze a wide array of data sources, including past box office statistics, rentals, and even pirated downloads. Taking advantage of vast data sets and insights from the generative AI models, the AI platform employed by Warner is set to free studio managers from laborious, iterative duties, empowering them to instead concentrate on delivering practical insights to drive key decisions regarding product packaging, distribution, marketing, and sales.\"}),/*#__PURE__*/e(\"p\",{children:\"This transition toward data-driven decision-making has the potential to revolutionize the movie industry, offering not only a deeper understanding of audience preferences but also the ability to make more strategic and profitable choices in film development.\"}),/*#__PURE__*/e(\"h3\",{children:\"Marketing\"}),/*#__PURE__*/e(\"p\",{children:\"Coca-Cola, a global beverage corporation, is adopting generative AI for innovative advertising campaigns. The company has taken significant steps to incorporate AI into its marketing strategies. For instance, the company created an advertisement set in a museum where famous artworks were brought to life through the use of OpenAI's DALL-E 2 generative image tool. This innovative approach not only added an element of interactivity to the ad but also showcased the creative potential of generative AI.\"}),/*#__PURE__*/e(\"p\",{children:\"Coca-Cola's proactive approach in integrating generative AI into its advertising strategies not only demonstrates its forward-thinking nature but also serves as an example of how AI can be a powerful tool in the hands of creative professionals.\"}),/*#__PURE__*/e(\"p\",{children:\"These are just a few examples of how enterprises are utilizing generative AI. As companies increasingly recognize the potential of these technologies, we can anticipate even more innovative applications in the future, further transforming the business landscape and opening up new possibilities for efficiency, creativity, and customer engagement.\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/e(\"p\",{children:\"Generative AI has proven to be transformative in the world of business, especially in tasks related to text generation. Its applications extend to content creation, language translation, market research, personalized recommendations, content moderation, and beyond.\"}),/*#__PURE__*/e(\"p\",{children:\"The field of generative AI is dynamic, with ongoing advancements and new models emerging every year. To stay competitive businesses must embrace continuous learning and adaptation to leverage the latest technologies effectively.\"}),/*#__PURE__*/e(\"p\",{children:\"Incorporating generative AI into business operations offers a competitive edge, enhances customer interactions, and streamlines various business functions. As generative AI advances, companies that strategically leverage this technology will be better positioned to succeed in a world that is becoming more and more dominated by data and automation.\"})]});export const richText6=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Large language models have wowed us all with their ability to invent a story, write a poem, or craft an email. But the impact of a model with creative writing capabilities reaches far beyond the wow factor. Businesses anticipate large gains in employee productivity and accelerated growth by leveraging generative AI for writing tasks. \"}),/*#__PURE__*/e(\"p\",{children:\"The question is: how do language models develop creative writing skills?\"}),/*#__PURE__*/e(\"p\",{children:\"The secret lies in designing interesting prompts for model training. Generative models rely on prompts, which means that the quality of training prompts can shape the quality of the model\u2019s output.\"}),/*#__PURE__*/t(\"p\",{children:[\"In a \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/tech-spec-for-your-own-llm/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"recent post\"})}),\" on technical specifications for building your own LLM, we shared the exciting news that Toloka is working on a joint project to build a large language model from scratch. In today\u2019s post, we\u2019ll share some first-hand details on how we collect the best prompts to help the model develop essential creative writing skills.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Aren\u2019t LLMs already good at writing?\"}),/*#__PURE__*/e(\"p\",{children:\"Large language models are innately good at generating texts. They are built to predict the next word and combine words into coherent narratives. But they need to learn how to follow the prompt and answer in a particular style \u2014 that\u2019s where fine-tuning comes in.\"}),/*#__PURE__*/e(\"p\",{children:\"To make LLMs better at creative writing, we need strong prompts that can elicit unique and interesting texts for fine-tuning the model. Here we run into two limitations: a lack of diverse datasets for training, and a lack of benchmarks for evaluating quality.\"}),/*#__PURE__*/e(\"h2\",{children:\"The dataset problem: synthetic vs organic data\"}),/*#__PURE__*/t(\"p\",{children:[\"There are many open datasets available for training, consisting of sets of prompts and good responses (these include \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/datasets/RyokoAI/ShareGPT52K\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"ShareGPT\"})}),\", WizardLM, OpenAssistant, Dolly, and Alpaca). However, most of these datasets are synthetic. In other words, the prompts and answers don\u2019t always express ideas that people are interested in. Some training datasets even use prompts generated by the models themselves.\"]}),/*#__PURE__*/e(\"p\",{children:\"The goal of our project is to help the language model develop authentic human-sounding writing skills, so we\u2019re generating an organic set of training data written fully by humans. While ours is certainly not the first LLM to use human-written prompt sets, we embarked on the project with a mission to make the scope and diversity of the data unique. We take full advantage of Toloka\u2019s global crowd to collect the most diverse prompts imaginable.\"}),/*#__PURE__*/e(\"h2\",{children:\"What makes a good prompt\"}),/*#__PURE__*/e(\"p\",{children:\"A good prompt is organic, and we invite Tolokers from all over the world to contribute prompts that draw on their own personal experience. They are assigned prompt writing tasks in different categories (essay, story, blog post, email, tweet, ad, movie script, and others) and they are free to write a prompt on any topic that interests them \u2014 as long as they follow our tips for the category in the task. Tolokers craft a prompt and submit it, without writing an answer for it. The text responses are later put together by experts in a separate project, and we\u2019ll share those details in an upcoming post.\"}),/*#__PURE__*/e(\"p\",{children:\"Our instructions encourage Tolokers to be creative and come up with prompts that contain plenty of context and aren\u2019t too trivial or generic (\u201CWrite a poem about love\u201D is a bad prompt). The guidelines for the task include examples of interesting prompts and not-so-interesting prompts. For instance, \u201CWrite a short story about a dog\u201D is not interesting enough to evoke a creative response, but \u201CWrite a story about a timid dog who traveled to outer space to save the planet\u201D provides a springboard for creativity.\"}),/*#__PURE__*/e(\"p\",{children:\"Here is an excerpt from the instructions and a submitted prompt.\"}),/*#__PURE__*/e(\"img\",{alt:\"Example of submitted prompt\",className:\"framer-image\",height:\"523\",src:\"https://framerusercontent.com/images/0Vt2ZOYi5cDVi68FTbN2jRwbhU.jpeg\",srcSet:\"https://framerusercontent.com/images/0Vt2ZOYi5cDVi68FTbN2jRwbhU.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/0Vt2ZOYi5cDVi68FTbN2jRwbhU.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/0Vt2ZOYi5cDVi68FTbN2jRwbhU.jpeg 1881w\",style:{aspectRatio:\"1881 / 1046\"},width:\"940\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Example of submitted prompt\"})}),/*#__PURE__*/e(\"h2\",{children:\"Creative writing prompts spotted in the wild\"}),/*#__PURE__*/e(\"p\",{children:\"The best prompts include full context, like the topic, the basic plot and characters, or even an outline with the desired structure. To help Tolokers with writer\u2019s block, we recommend making it personal \u2014 maybe there\u2019s a topic that\u2019s been on their mind lately, their child asked for a story, or they need to write something for work. Anything that is relevant to them or someone they know is a great subject for a prompt.\"}),/*#__PURE__*/e(\"p\",{children:\"Here are some examples of writing prompts recently submitted by Tolokers.\"}),/*#__PURE__*/e(\"img\",{alt:\"Examples of good prompts\",className:\"framer-image\",height:\"729\",src:\"https://framerusercontent.com/images/A4SDgs2RhofqL4ZJyveZYtmQKiY.jpeg\",srcSet:\"https://framerusercontent.com/images/A4SDgs2RhofqL4ZJyveZYtmQKiY.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/A4SDgs2RhofqL4ZJyveZYtmQKiY.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/A4SDgs2RhofqL4ZJyveZYtmQKiY.jpeg 1920w\",style:{aspectRatio:\"1920 / 1458\"},width:\"960\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Examples of good prompts\"})}),/*#__PURE__*/e(\"h2\",{children:\"The evaluation problem: how to verify prompt quality\"}),/*#__PURE__*/e(\"p\",{children:\"Now let\u2019s consider the second roadblock: checking prompt quality. Benchmarks that evaluate the quality of models generally compare a text answer to the \u201Cgolden\u201D answer, word for word. But this isn\u2019t a meaningful way to measure the quality of creative texts, which should be unique by definition.\"}),/*#__PURE__*/e(\"p\",{children:\"The only reasonable way to judge creativity is to collect the opinions of real people. Unfortunately, it\u2019s prohibitively expensive and time-consuming to read through thousands of texts. We\u2019ve implemented a better solution: an automated human verification pipeline using the Toloka crowd.\"}),/*#__PURE__*/e(\"p\",{children:\"Tolokers who perform verification tasks are given a prompt written by another person, along with the guidelines. They decide if the prompt follows the guidelines or not \u2014 a simple question of \u201Cgood\u201D or \u201Cbad\u201D. The key here is to make the guidelines as clear and specific as possible, with plenty of examples of what makes a prompt good or bad.\"}),/*#__PURE__*/e(\"img\",{alt:\"Example of a verification task\",className:\"framer-image\",height:\"857\",src:\"https://framerusercontent.com/images/zNHMarfPkzBe2bOXrTxHcVUQTaw.jpeg\",srcSet:\"https://framerusercontent.com/images/zNHMarfPkzBe2bOXrTxHcVUQTaw.jpeg?scale-down-to=1024 937w,https://framerusercontent.com/images/zNHMarfPkzBe2bOXrTxHcVUQTaw.jpeg 1569w\",style:{aspectRatio:\"1569 / 1714\"},width:\"784\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Example of a verification task\"})}),/*#__PURE__*/e(\"p\",{children:\"Since this works like a binary text classification task, we can use our proven quality control methods in the pipeline to make sure verification is accurate. One of our core methods is overlap, meaning we assign multiple people to verify the same prompt and then aggregate their answers to get a reliable final verdict. We use the prompts that \u201Cpass\u201D for model training and discard the ones that are rejected. With our large crowd of Tolokers, the process is fast and efficient.\"}),/*#__PURE__*/e(\"h2\",{children:\"Why we prioritize prompt diversity\"}),/*#__PURE__*/e(\"p\",{children:\"We\u2019ve collected over 6000 prompts for creative writing. But what\u2019s impressive is the diversity in this dataset. Thousands of Tolokers from 69 different countries with a multitude of backgrounds and cultural contexts have contributed ideas that are meaningful to them.\"}),/*#__PURE__*/e(\"p\",{children:\"A model that is trained on diverse data can emulate a broader range of human experiences and expectations, making it less biased and more helpful to real people in future applications.\"}),/*#__PURE__*/t(\"p\",{children:[\"Looking to collect prompts for your own LLM? Tap into the Toloka crowd for a global perspective. \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/large-language-models/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Reach out\"})}),\" to discuss the ideal pipeline for your project.\"]})]});export const richText7=/*#__PURE__*/t(o.Fragment,{children:[/*#__PURE__*/e(\"h2\",{children:\"How to build a pipeline to achieve superhuman quality  \"}),/*#__PURE__*/e(\"p\",{children:\"You may have heard that Large Language Models (LLMs) are faster, cheaper, and better than humans at text annotation. Does this mean we no longer need human data labeling? Not exactly. \"}),/*#__PURE__*/e(\"p\",{children:\"Based on our experience using LLMs on real-world text annotation projects, even the latest state-of-the-art models aren\u2019t meeting quality expectations. What\u2019s more, these models aren\u2019t always cheaper than data labeling with human annotators.\"}),/*#__PURE__*/t(\"p\",{children:[\"But we\u2019ve found that it is possible to elevate data quality by using an \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/labeling-with-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"optimal mix of human and LLM labeling\"})}),\".\"]}),/*#__PURE__*/e(\"h2\",{children:\"Comparing the quality of LLMs and human labeling\"}),/*#__PURE__*/e(\"p\",{children:\"There are a growing number of general-use LLMs publicly available, and they all belong to one of two camps: open source or closed API. Open source models are generally much cheaper to run.\"}),/*#__PURE__*/t(\"p\",{children:[\"You can see comparisons of overall model performance on Meta\u2019s \",/*#__PURE__*/e(a,{href:\"https://ai.meta.com/llama/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"benchmarks table\"})}),\", in \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/pdf/2307.09288.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"this paper\"})}),\", and on the \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"LMSYS leaderboard\"})}),\". However, most evaluation projects are based on open source datasets. To get a clear picture of LLM performance, we need to compare output on real-world projects as well.\"]}),/*#__PURE__*/e(\"p\",{children:\"We\u2019ve been testing multiple LLMs on our own data labeling projects and comparing them to human labeling with a crowd of trained annotators. To assess the quality of both humans and models, we compare their labels to ground truth labels prepared by experts.\"}),/*#__PURE__*/e(\"p\",{children:\"For most of the projects we run, we see good results from two models in particular:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"GPT-4 from OpenAI (closed API)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Llama 2 from Meta (open source) \"})})]}),/*#__PURE__*/e(\"h2\",{children:\"The dilemma of optimizing costs vs quality\"}),/*#__PURE__*/e(\"p\",{children:\"If we\u2019re talking about an open source model (Llama 2) where we only pay for GPU usage, data labeling might be hundreds of times cheaper than human crowd labeling. But what\u2019s the tradeoff? It all depends on the complexity of the task.\"}),/*#__PURE__*/e(\"p\",{children:\"Our experiments have achieved near-human accuracy with Llama 2 on sentiment analysis, spam detection, and a few other types of tasks. The inference price is generally low, but there are some other expenses involved. You need to have the infrastructure to run fine-tuning and inference, and you need to collect a human-labeled dataset for fine-tuning at the outset.\"}),/*#__PURE__*/e(\"p\",{children:\"GPT-4 provides acceptable results right out of the gate. You just need to write a detailed prompt with task instructions and examples in text format. However, the cost of inference is either more expensive or only slightly cheaper than human labeling \u2014 and the quality is lower.\"}),/*#__PURE__*/e(\"p\",{children:\"The good news is you don\u2019t have to choose between using a model or using human annotation, and you don\u2019t have to sacrifice quality to gain efficiency. We get the best results when we maximize the capabilities of human labeling and LLM labeling at the same time.\"}),/*#__PURE__*/t(\"p\",{children:[\"By carefully analyzing output, we often find subsets of data where a fine-tuned LLM performs better than humans. We can take advantage of this strength to achieve \",/*#__PURE__*/e(\"strong\",{children:\"overall quality that is better than human-only or LLM-only labeling\"}),\".\"]}),/*#__PURE__*/e(\"p\",{children:\"Does this mean we have a silver bullet to enhance quality without spending more money? Absolutely.\"}),/*#__PURE__*/e(\"h2\",{children:\"The silver bullet: intelligent hybrid pipelines\"}),/*#__PURE__*/e(\"p\",{children:\"Toloka uses hybrid pipelines, meaning we add an LLM to the data labeling pipeline alongside humans. When done correctly, this approach can optimize costs while achieving unprecedented quality.\"}),/*#__PURE__*/e(\"p\",{children:\"So how do we structure a hybrid pipeline?\"}),/*#__PURE__*/t(\"p\",{children:[\"When LLMs output a label, they can also output the \",/*#__PURE__*/e(a,{href:\"https://www.kdnuggets.com/the-quest-for-model-confidence-can-you-trust-a-black-box\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"confidence level\"})}),\". We can use this information in a pipeline that combines LLM and human labeling. The approach is simple: the LLM labels the data, and labels with low confidence are sent to the crowd for relabeling.\"]}),/*#__PURE__*/e(\"p\",{children:\"We can change the confidence threshold to adjust the amount of data that is relabeled by humans and control the quality of the final labels.\"}),/*#__PURE__*/e(\"img\",{alt:\"Quality vs relabeling ratio. Image by author\",className:\"framer-image\",height:\"796\",src:\"https://framerusercontent.com/images/52ByVvLoy5DUTtqYByxH8505U.jpeg\",srcSet:\"https://framerusercontent.com/images/52ByVvLoy5DUTtqYByxH8505U.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/52ByVvLoy5DUTtqYByxH8505U.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/52ByVvLoy5DUTtqYByxH8505U.jpeg?scale-down-to=2048 2048w,https://framerusercontent.com/images/52ByVvLoy5DUTtqYByxH8505U.jpeg 2304w\",style:{aspectRatio:\"2304 / 1592\"},width:\"1152\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Quality vs relabeling ratio. Image by author  \"})}),/*#__PURE__*/e(\"p\",{children:\"For typical tasks, the more we use human labeling, the higher the cost. The trick is to find the right threshold for model confidence to optimize for quality or cost, depending on our goal. We can choose any point on this curve and achieve the cost-quality trade-off that is needed for a specific task.\"}),/*#__PURE__*/e(\"p\",{children:\"In some cases, quality optimization with LLMs just doesn\u2019t work. This usually happens on complex tasks where the model\u2019s quality lags far behind human quality. We can still use the model for cost optimization and get results that are very close to human accuracy, as shown below.\"}),/*#__PURE__*/e(\"img\",{alt:\" optimization and get results that are very close to human accuracy, as shown below.\",className:\"framer-image\",height:\"796\",src:\"https://framerusercontent.com/images/GNrOOxpAx958DaxIUsMSl6O5nI.jpeg\",srcSet:\"https://framerusercontent.com/images/GNrOOxpAx958DaxIUsMSl6O5nI.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/GNrOOxpAx958DaxIUsMSl6O5nI.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/GNrOOxpAx958DaxIUsMSl6O5nI.jpeg?scale-down-to=2048 2048w,https://framerusercontent.com/images/GNrOOxpAx958DaxIUsMSl6O5nI.jpeg 2256w\",style:{aspectRatio:\"2256 / 1593\"},width:\"1128\"}),/*#__PURE__*/e(\"h2\",{children:\"Examples of cost-optimized pipelines\"}),/*#__PURE__*/e(\"p\",{children:\"Here are some real-life results of using cost-optimized hybrid pipelines, compared to human-only and LLM-only workflows.\"}),/*#__PURE__*/e(\"img\",{alt:\"*Cost reduction includes only human labor and inference costs. It does not reflect overall savings per project.\",className:\"framer-image\",height:\"561\",src:\"https://framerusercontent.com/images/SD0iuUya8s1xlMa72goFZ8W2W7g.png\",srcSet:\"https://framerusercontent.com/images/SD0iuUya8s1xlMa72goFZ8W2W7g.png?scale-down-to=512 512w,https://framerusercontent.com/images/SD0iuUya8s1xlMa72goFZ8W2W7g.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/SD0iuUya8s1xlMa72goFZ8W2W7g.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/SD0iuUya8s1xlMa72goFZ8W2W7g.png 2714w\",style:{aspectRatio:\"2714 / 1122\"},width:\"1357\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"*Cost reduction includes only human labor and inference costs. It does not reflect overall savings per project.\"})}),/*#__PURE__*/e(\"p\",{children:\"The potential for optimization strongly depends on the complexity of the task. Naturally, the further the model's output lags behind human quality, the less we can automate.\"}),/*#__PURE__*/e(\"h2\",{children:\"Examples of quality-optimized pipelines\"}),/*#__PURE__*/e(\"p\",{children:\"This table shows results on quality-optimized pipelines, where we leveraged the LLM to achieve better-than-human quality.\"}),/*#__PURE__*/e(\"img\",{alt:\"*Cost reduction includes only human labor and inference costs. It does not reflect overall savings per project.\",className:\"framer-image\",height:\"409\",src:\"https://framerusercontent.com/images/w4VyUvnKVXBNpKeGJjPqj8YdSs.png\",srcSet:\"https://framerusercontent.com/images/w4VyUvnKVXBNpKeGJjPqj8YdSs.png?scale-down-to=512 512w,https://framerusercontent.com/images/w4VyUvnKVXBNpKeGJjPqj8YdSs.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/w4VyUvnKVXBNpKeGJjPqj8YdSs.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/w4VyUvnKVXBNpKeGJjPqj8YdSs.png 2714w\",style:{aspectRatio:\"2714 / 818\"},width:\"1357\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"*Cost reduction includes only human labor and inference costs. It does not reflect overall savings per project.\"})}),/*#__PURE__*/e(\"p\",{children:\"On these tasks, the optimized hybrid pipeline performed impressively well compared to humans alone, while also reducing costs.\"}),/*#__PURE__*/e(\"p\",{children:\"Here's how we achieve superhuman accuracy:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Fine-tune models using high-quality datasets labeled by domain experts.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Strategically choose the confidence threshold for labeling with LLMs, and use human labeling to close any gaps.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Identify the subset of data where LLM output is more accurate than human labeling \u2014 the bigger the subset, the more cost savings are possible while meeting high quality standards.\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"Human + LLM pipelines in production\"}),/*#__PURE__*/e(\"p\",{children:\"Reliable quality is our number one priority, and we build quality control steps into every data labeling pipeline that goes into production. We check up to 5% of the final labels \u2014 both LLM output and human output \u2014 using validation by experts. We then use the results to build metrics that help us continually monitor the overall quality of the pipeline.\"}),/*#__PURE__*/e(\"p\",{children:\"So how do we mitigate quality issues? If quality drops, the standard solution is to add more data and fine-tune the model again. Sometimes the data keeps getting more complex and we don\u2019t see improvement in the model\u2019s output. In this case, we can adjust the confidence threshold for the model so that a smaller percentage of data is labeled by the model, and more data is labeled by humans.\"}),/*#__PURE__*/e(\"img\",{alt:\"Hybrid data labeling pipeline. Image by author\",className:\"framer-image\",height:\"1342\",src:\"https://framerusercontent.com/images/UUKelMvxDapOT2lzjvKC70iu2A.jpeg\",srcSet:\"https://framerusercontent.com/images/UUKelMvxDapOT2lzjvKC70iu2A.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/UUKelMvxDapOT2lzjvKC70iu2A.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/UUKelMvxDapOT2lzjvKC70iu2A.jpeg?scale-down-to=2048 2048w,https://framerusercontent.com/images/UUKelMvxDapOT2lzjvKC70iu2A.jpeg?scale-down-to=4096 4096w,https://framerusercontent.com/images/UUKelMvxDapOT2lzjvKC70iu2A.jpeg 4172w\",style:{aspectRatio:\"4172 / 2684\"},width:\"2086\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Hybrid data labeling pipeline. Image by author\"})}),/*#__PURE__*/e(\"p\",{children:\"With the right expertise for fine-tuning and the perfect threshold to balance LLM and human labeling, hybrid pipelines can produce exceptional quality in many cases.\"}),/*#__PURE__*/e(\"h2\",{children:\"Preparing your own pipeline: paving the way to success\"}),/*#__PURE__*/e(\"p\",{children:\"To run your own hybrid data labeling pipeline that leverages LLMs and human insight, you\u2019ll need to lay the groundwork by covering 4 essential steps:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"1.  A golden set\"}),\": To improve the quality of LLM output on your specific task, you need an accurate expert-labeled dataset for fine-tuning. The more data you have and the higher the quality, the better your fine-tuned model will perform \u2014 and the more you\u2019ll be able to automate data labeling. You can rely on qualified annotators from the \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/data-labeling-platform/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Toloka crowd\"})}),\" to collect high-quality training data.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"2.  Infrastructure for model training and inference\"}),\" and ML expertise for working with LLMs. Effective GPU utilization is crucial to make model labeling as cost-effective as possible. You will need:\"]}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Cloud instance with multiple GPUs\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Code for model fine-tuning and inference\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Storage and versioning of model weights\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Tools for model validation and quality monitoring\"})})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"3.  An annotator workforce\"}),\" available on demand to handle labels with low confidence or complex tasks. Toloka\u2019s global crowd is available 24/7 with annotators in every time zone.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"4. Quality control strategy\"}),\". Monitor the quality of output with ongoing human verification. Incorrect labels are sent for relabeling. At Toloka, we use advanced quality control techniques for human and LLM labeling, with continuous quality monitoring.\"]}),/*#__PURE__*/e(\"p\",{children:\"Toloka can help you in every stage of the AI development process. From finding the right LLM for your task, to taking care of fine-tuning, and designing a hybrid pipeline tailored to your needs \u2014 our team is here to support you every step of the way. The decision to optimize for quality or cost is always up to you!\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(a,{href:\"https://toloka.ai/talk-to-us/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Talk to Toloka's AI experts\"})})})]});\nexport const __FramerMetadata__ = {\"exports\":{\"richText7\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText5\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText6\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText4\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText3\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText1\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText2\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"__FramerMetadata__\":{\"type\":\"variable\"}}}"],
  "mappings": "+LAAsJ,IAAMA,EAAsBC,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,8MAA8M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sJAAsJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uCAAuC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mIAAmI,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,qBAAqB,UAAU,eAAe,OAAO,KAAK,IAAI,sEAAsE,OAAO,sKAAsK,MAAM,CAAC,YAAY,UAAU,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sVAAsV,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,cAAc,UAAU,eAAe,OAAO,KAAK,IAAI,sEAAsE,MAAM,CAAC,YAAY,UAAU,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sZAAsZ,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,eAAe,UAAU,eAAe,OAAO,KAAK,IAAI,uEAAuE,MAAM,CAAC,YAAY,UAAU,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uOAAuO,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,gBAAgB,UAAU,eAAe,OAAO,KAAK,IAAI,qEAAqE,MAAM,CAAC,YAAY,UAAU,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4gBAAkgB,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,yBAAyB,UAAU,eAAe,OAAO,KAAK,IAAI,uEAAuE,OAAO,wKAAwK,MAAM,CAAC,YAAY,UAAU,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uQAAuQ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iNAAiN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yEAAyE,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6GAA6G,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2GAA2G,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uGAAuG,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2MAA2M,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qHAAqH,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8XAA8X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uHAAuH,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6CAA6C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6XAA6X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+ZAA+Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iZAAiZ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,oCAAoC,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qUAAqU,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kOAAkO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gNAA2M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6hBAA6hB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kFAAkF,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gHAAgH,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeC,EAAuBL,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,4UAAuU,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,yGAAsHE,EAAEC,EAAE,CAAC,KAAK,4DAA4D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,mCAAmC,CAAC,CAAC,CAAC,EAAE,4VAA4V,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,mOAAmO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8ZAA8Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uTAAkT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ySAA+R,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8CAA8C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0KAA0K,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,+CAA+C,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,iDAAiD,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6QAA6Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wYAAwY,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,sBAAsB,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,qBAAqB,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,oBAAoB,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0aAAqa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wQAAwQ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8VAAoV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mdAAmd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uhBAA6gB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,0OAAuPE,EAAEC,EAAE,CAAC,KAAK,oEAAoE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,2KAA2K,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,8BAA8B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,kBAA+BE,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAE,mJAAmJ,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAE,+DAA4EA,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,EAAE,+NAA+N,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,UAAU,CAAC,EAAE,oVAAoV,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,yDAAsEE,EAAE,KAAK,CAAC,CAAC,EAAE,kWAAwV,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qbAA2a,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0JAA0J,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,EAAE,mbAAmb,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,EAAE,8KAA8K,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,EAAE,2QAA2Q,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sDAAsD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gMAAgM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,EAAE,kaAAka,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,EAAE,sSAAsS,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,EAAE,sjBAAsjB,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iCAAiC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,qDAAqD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4BAA4B,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,4DAA4D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,8EAA8E,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wEAAwE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeE,EAAuBN,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,2PAA2P,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iBAA8BE,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oQAAoQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kSAAkS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8WAA8W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qIAAqI,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4GAA4G,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4EAA4E,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,4HAA4H,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,wEAAwE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gbAAgb,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,UAAU,CAAC,EAAE,kKAA+KA,EAAE,SAAS,CAAC,SAAS,UAAU,CAAC,EAAE,iLAAiL,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,wLAAwL,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,gBAA6BE,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,EAAE,oPAAoP,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,2FAAwGE,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,EAAE,kKAAkK,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,4PAA4P,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,yCAAsDE,EAAE,SAAS,CAAC,SAAS,qCAAqC,CAAC,EAAE,sEAAsE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wGAAwG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0LAA0L,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,wDAAwD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,2DAA2D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,kCAAkC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8EAA8E,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,otBAAotB,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,uBAAuB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,6TAA0UF,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,oEAAoE,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,0DAA0D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,oNAAoN,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,uDAAoEE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,MAAM,CAAC,CAAC,CAAC,EAAE,qEAAkFF,EAAEC,EAAE,CAAC,KAAK,4EAA4E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uCAAuC,CAAC,CAAC,CAAC,EAAE,6MAA6M,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAC,wBAAqCE,EAAEC,EAAE,CAAC,KAAK,wGAAwG,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,gIAAgI,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,4EAA4E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAE,IAAiBF,EAAEC,EAAE,CAAC,KAAK,kCAAkC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAE,QAAqBF,EAAEC,EAAE,CAAC,KAAK,oDAAoD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sCAAsC,CAAC,CAAC,CAAC,EAAE,0TAA0T,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,4EAA4E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAE,ofAAof,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,kCAAkC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAE,kdAAkd,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,oDAAoD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yCAAyC,CAAC,CAAC,CAAC,EAAE,qJAAkKF,EAAEC,EAAE,CAAC,KAAK,uDAAuD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAE,oKAAoK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,6GAA0HE,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,EAAE,+DAA4EF,EAAEC,EAAE,CAAC,KAAK,iDAAiD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,uKAAoLF,EAAEC,EAAE,CAAC,KAAK,oDAAoD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,4KAAyLF,EAAEC,EAAE,CAAC,KAAK,iEAAiE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,IAAiBF,EAAEC,EAAE,CAAC,KAAK,8CAA8C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,QAAqBF,EAAEC,EAAE,CAAC,KAAK,mDAAmD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uCAAuC,CAAC,CAAC,CAAC,EAAE,4DAA4D,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mUAAmU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mJAAmJ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wRAAwR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,6JAA0KE,EAAEC,EAAE,CAAC,KAAK,+BAA+B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,kBAAkB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeG,EAAuBP,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,ogBAA+f,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+sBAA0sB,CAAC,EAAeF,EAAE,aAAa,CAAC,SAAS,CAAcE,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,kOAAkO,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2CAAsC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4uBAAouB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iTAAuS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6dAA6d,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oaAAoa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4LAAkL,CAAC,EAAeF,EAAE,aAAa,CAAC,SAAS,CAAcE,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,yHAAoH,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qCAAgC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6PAA6P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yDAAyD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gcAA2b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mYAAoY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,saAAsa,CAAC,EAAeF,EAAE,aAAa,CAAC,SAAS,CAAcE,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,oHAA+G,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2CAAsC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wDAAqEE,EAAEC,EAAE,CAAC,KAAK,yGAAyG,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4BAA4B,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeI,EAAuBR,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,oZAA+Y,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,uCAAoDE,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAE,iTAAiT,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,obAAicE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,qDAAqD,CAAC,CAAC,CAAC,EAAE,iMAA4L,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,kDAAkD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,kFAA+FE,EAAE,SAAS,CAAC,SAAS,gCAAgC,CAAC,EAAE,8QAA8Q,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iCAA8CE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,4PAA4P,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2XAA2X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mbAAmb,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,2BAAwCE,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,EAAE,idAAid,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wDAAqEE,EAAE,SAAS,CAAC,SAAS,eAAe,CAAC,EAAE,6TAA6T,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+RAA+R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8TAA8T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4YAA4Y,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6PAA6P,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ukBAAukB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qDAAqD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2YAA2Y,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kWAAkW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0NAA0N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+VAA+V,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wXAAwX,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+XAA4YE,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,mgBAAmgB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qRAAqR,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wCAAwC,CAAC,EAAE,gPAAgP,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,6RAA6R,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,EAAE,qQAAqQ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,sZAAsZ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,EAAE,+SAA+S,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+QAA+Q,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iHAAiH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sFAAsF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iFAAiF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iHAAiH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kMAAkM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sTAAsT,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sJAAsJ,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uEAAuE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uIAAuI,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kFAAkF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6GAA6G,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uJAAuJ,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0GAA0G,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6HAA6H,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gZAAgZ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8BAA8B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAE,oUAAoU,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,0CAA0C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,igBAAigB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,gEAAgE,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yiCAAyiC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,+CAA+C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8bAA8b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0SAA0S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uaAAua,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,EAAE,mYAAoY,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wSAAwS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,iEAAiE,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oTAAoT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4PAA4P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kJAAkJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,cAAc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8pBAA8pB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uZAAuZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wWAAwW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,cAAc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ybAAyb,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qWAAqW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,y1BAAy1B,CAAC,CAAC,CAAC,CAAC,EAAeO,EAAuBT,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,oTAAoT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+TAA+T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yUAAyU,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2OAA2O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+bAA+b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oQAAoQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wcAAwc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oUAAoU,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,8CAA8C,CAAC,EAAE,yOAAyO,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gEAAgE,CAAC,EAAE,wPAAwP,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,6CAA6C,CAAC,EAAE,oXAAoX,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAyB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oPAAoP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4iBAA4iB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wPAAwP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ieAAie,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mhBAAmhB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mcAAmc,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qUAAqU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mSAAmS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0NAA0N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qaAAqa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6OAA6O,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wEAAwE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+LAA+L,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,miBAAmiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8cAA8c,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gUAAgU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAoX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wMAAwM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sOAAsO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mVAAmV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iTAAiT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uPAAuP,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,2MAA2M,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,EAAE,6MAA6M,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,+IAA+I,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,EAAE,8FAA8F,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,oJAAoJ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8XAA8X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6NAA6N,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mLAAmL,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+NAA+N,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2CAA2C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yVAAyV,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wOAAwO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0LAA0L,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sKAAsK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,siBAAsiB,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2IAA2I,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2JAA2J,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iIAAiI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2eAA2e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qYAAqY,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mPAAmP,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mKAAmK,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uOAAuO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8eAA8e,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yPAAyP,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2NAA2N,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oMAAoM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4UAA4U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mYAAmY,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gKAAgK,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0IAA0I,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gLAAgL,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+CAA+C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAA6T,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,yTAAyT,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,+LAA+L,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,mNAAmN,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,kJAAkJ,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,qMAAqM,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kLAAkL,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+WAA+W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kZAAkZ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,maAAma,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gTAAgT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6lBAA6lB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oQAAoQ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wfAAwf,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAAsP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6VAA6V,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2QAA2Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sOAAsO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+VAA+V,CAAC,CAAC,CAAC,CAAC,EAAeQ,EAAuBV,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,kVAAkV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0EAA0E,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4MAAuM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,QAAqBE,EAAEC,EAAE,CAAC,KAAK,qDAAqD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,4UAAkU,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,2CAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kRAAwQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qQAAqQ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wHAAqIE,EAAEC,EAAE,CAAC,KAAK,sDAAsD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,kRAA6Q,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,ycAA+b,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wmBAA8lB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,siBAAmgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kEAAkE,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,8BAA8B,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8CAA8C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2bAAua,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2EAA2E,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,2BAA2B,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,0QAA0Q,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sDAAsD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAAyS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2SAAiS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iXAAwV,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,iCAAiC,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,4KAA4K,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0eAAge,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uRAA6Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0LAA0L,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,oGAAiHE,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,kDAAkD,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeO,EAAuBX,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAS,yDAAyD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0LAA0L,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kQAAmP,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gFAAwFE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uCAAuC,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,kDAAkD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8LAA8L,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,uEAA+EE,EAAEC,EAAE,CAAC,KAAK,6BAA6B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAE,QAAqBF,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,gBAA6BF,EAAEC,EAAE,CAAC,KAAK,gEAAgE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,EAAE,6KAA6K,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,uQAAkQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qFAAqF,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kCAAkC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qPAA2O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8WAA8W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6RAAwR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iRAAuQ,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,sKAAmLE,EAAE,SAAS,CAAC,SAAS,qEAAqE,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oGAAoG,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iDAAiD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kMAAkM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2CAA2C,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,sDAAmEE,EAAEC,EAAE,CAAC,KAAK,qFAAqF,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAE,yMAAyM,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,8IAA8I,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,+CAA+C,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gTAAgT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mSAAyR,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,uFAAuF,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0HAA0H,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,kHAAkH,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,iHAAiH,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+KAA+K,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2HAA2H,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,kHAAkH,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,iHAAiH,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gIAAgI,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4CAA4C,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yEAAyE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iHAAiH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0LAAqL,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+WAAqW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mZAAyY,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,iDAAiD,UAAU,eAAe,OAAO,OAAO,IAAI,uEAAuE,OAAO,mcAAmc,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uKAAuK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wDAAwD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4JAAuJ,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,gVAAmVA,EAAEC,EAAE,CAAC,KAAK,4CAA4C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,yCAAyC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qDAAqD,CAAC,EAAE,oJAAoJ,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mCAAmC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0CAA0C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yCAAyC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mDAAmD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,8JAAyJ,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,EAAE,kOAAkO,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mUAA8T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAC7vsIQ,EAAqB,CAAC,QAAU,CAAC,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,SAAW,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,mBAAqB,CAAC,KAAO,UAAU,CAAC,CAAC",
  "names": ["richText", "u", "x", "p", "Link", "motion", "richText1", "richText2", "richText3", "richText4", "richText5", "richText6", "richText7", "__FramerMetadata__"]
}