{
  "version": 3,
  "sources": ["ssg:https://framerusercontent.com/modules/YVgynkVt96a0ES9Gk8yz/xMdX6bSnyiOlePRFzTMI/eo4RAmtig-7.js"],
  "sourcesContent": ["import{jsx as e,jsxs as t}from\"react/jsx-runtime\";import{Link as n}from\"framer\";import{motion as i}from\"framer-motion\";import*as a from\"react\";export const richText=/*#__PURE__*/t(a.Fragment,{children:[/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"600\",src:\"https://framerusercontent.com/images/rnOhtC0RlpU6bvwG3G3CNW1WQ.png\",srcSet:\"https://framerusercontent.com/images/rnOhtC0RlpU6bvwG3G3CNW1WQ.png?scale-down-to=512 512w,https://framerusercontent.com/images/rnOhtC0RlpU6bvwG3G3CNW1WQ.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/rnOhtC0RlpU6bvwG3G3CNW1WQ.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/rnOhtC0RlpU6bvwG3G3CNW1WQ.png 2400w\",style:{aspectRatio:\"2400 / 1201\"},width:\"1200\"}),/*#__PURE__*/e(\"p\",{children:\"LLMs show strong coding abilities, but they have mostly been tested on isolated, small-size problems, like the basic Python tasks in the well-known HumanEval and MBPP benchmarks. This leaves their proficiency in realistic, large-scale projects still unexplored. As these models become more integrated into professional coding workflows, the big question is: Are they truly ready to assist in real-world software development?\\xa0\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://www.swebench.com/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"SWE-bench\"})}),\" was created to address this gap, serving as an industry-standard benchmark used by tech leaders like OpenAI, Anthropic, DeepSeek, and Amazon to test models in realistic coding environments. However, SWE-bench has significant limitations that skew evaluations, making it difficult to assess an LLM\u2019s true capabilities.\"]}),/*#__PURE__*/e(\"p\",{children:\"In this article, we\u2019ll break down the shortcomings of SWE-bench, how to clean and refine the dataset to ensure fair, reproducible assessments, and the next steps to develop an even more comprehensive, accurate, and scalable benchmark.\"}),/*#__PURE__*/e(\"h2\",{children:\"What is SWE-bench?\"}),/*#__PURE__*/t(\"p\",{children:[\"SWE-bench was designed to address a recurring dilemma: \",/*#__PURE__*/e(\"strong\",{children:\"Can LLMs solve real GitHub issues as effectively as human developers?\"}),\" Unlike coding competition tasks that focus on isolated problems, real-world software development entails debugging large codebases, analyzing issue reports, and making coordinated changes across multiple files. SWE-bench requires models to understand an issue within the broader context of a software project, implement fixes, and verify the solution through testing.\\xa0\"]}),/*#__PURE__*/e(\"p\",{children:\"The SWE-bench dataset is built from real GitHub issues and their corresponding bug-fixing commits. Each sample includes a problem description or feature request, alongside the pull request containing the code changes and unit tests that address the issue. LLMs are tasked with using the description context to generate patches that resolve the issues by modifying the codebase. If the unit tests pass after implementing the solution, the sample is marked as correctly solved. This data collection approach is a more realistic way to capture the complexities and nuances of maintaining large-scale software projects.\"}),/*#__PURE__*/e(\"p\",{children:\"SWE-bench evaluations are in high demand among model developers. However, the benchmark needs to be cleaned up and extended with Dockerfiles before it\u2019s ready to apply in auto-evals.\\xa0\"}),/*#__PURE__*/e(\"h2\",{children:\"Where the current benchmark falls short\"}),/*#__PURE__*/e(\"p\",{children:\"Although the SWE-bench provides a strong foundation, it does have some shortcomings.\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Unfit tests\"}),\". Some tests are irrelevant or overly specific, leading to many false negatives. For example, a test might reject a correct solution because it slightly varies from the reference solution.\\xa0\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Vague or ambiguous problem descriptions.\"}),\" The tasks in this dataset aren't always clear, making it difficult for the model to understand the problem or determine what needs to be solved. This ambiguity causes models to generate incorrect solutions due to misinterpreting requirements.\\xa0\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Inconsistent development environments\"}),\". A patch that works in one setting might fail in another\u2014not due to flaws in the solution, but because of differences in setup (specific dependencies, configurations, and system settings). These inconsistencies can lead to misleading test results, where failures stem from mismatched environments rather than the model's actual performance.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Outdated dataset\"}),\". The latest pull request in SWE-bench is dated 2023. For Python, this is ancient\u2014leading to irrelevant evaluations or misleading claims about model performance based on older tasks that don\u2019t reflect the modern coding landscape. The benchmark needs to be updated to keep up with evolving best practices, updates, and deprecations.\\xa0\\xa0\"]})})]}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"729\",src:\"https://framerusercontent.com/images/b5yCbJs2BFFtqCmQ71mQsppWHw.png\",srcSet:\"https://framerusercontent.com/images/b5yCbJs2BFFtqCmQ71mQsppWHw.png?scale-down-to=512 512w,https://framerusercontent.com/images/b5yCbJs2BFFtqCmQ71mQsppWHw.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/b5yCbJs2BFFtqCmQ71mQsppWHw.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/b5yCbJs2BFFtqCmQ71mQsppWHw.png 2560w\",style:{aspectRatio:\"2560 / 1458\"},width:\"1280\"}),/*#__PURE__*/e(\"ul\",{children:/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Lack of automated environment replication:\"}),\" The main problem with SWE-bench is that we can't use it for automatic evaluation without manually setting up the environment for each repository first, which is very resource-intensive and time-consuming (\",/*#__PURE__*/e(n,{href:\"https://arxiv.org/html/2410.03859v1#:~:text=3.%20Set%20up%20the%20environment%20for%20each%20repository\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"research \"})}),\"shows that manually creating functional environments for each repository can take an average of 10 hours). One possible solution is creating a Dockerfile for each data point, which still requires significant time and effort, with obstacles like missing dependencies, unsupported package versions, or conflicts between old and new system libraries.\\xa0\"]})})}),/*#__PURE__*/t(\"p\",{children:[\"To put things in perspective, the current best-performing agent, powered by Claude 3.5, can solve only \",/*#__PURE__*/e(n,{href:\"https://www.swebench.com/#test\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"29% of the issues\"})}),\". Acknowledging the benchmark\u2019s limitations, we must ask ourselves: Are the models really that far from independently handling compound coding tasks, or are the evaluation standards skewed?\"]}),/*#__PURE__*/e(\"p\",{children:\"The weaknesses in SWE-bench make model assessments unreliable. Yet, there are ways to address the shortcomings and yield a benchmark that truly mirrors the challenges of real-world software development. My team at Toloka took the first step to upgrade SWE-bench with a thorough cleanup, and laid out plans for future work that will lead to a new and improved benchmark.\"}),/*#__PURE__*/e(\"h2\",{children:\"Cleaning up the SWE-bench dataset\"}),/*#__PURE__*/e(\"p\",{children:\"Developing benchmarks that evaluate code quality and problem-solving skills requires a deep understanding of how software works across different environments. Drawing on deep expertise, the Toloka team collaborated with coding experts in our network of data annotators, giving them stringent guidelines to refine the dataset and create structured, multi-turn code-editing demonstrations. The coding experts assessed training samples based on problem specification, test coverage, and correctness\u2014carefully annotating or rejecting samples to maintain data quality.\"}),/*#__PURE__*/t(\"p\",{children:[\"Our \",/*#__PURE__*/e(\"strong\",{children:\"two-step approach\"}),\" is designed to filter out low-quality samples and enhance benchmark reliability.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Step 1. Evaluate the quality of the samples.\\xa0\"})}),/*#__PURE__*/e(\"p\",{children:\"The first step is to identify samples that contain unsuitable tests or confusing descriptions. Annotators rate each sample on three criteria: clarity of the issue, test relevance, and difficulty.\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Clarity\"}),\" looks at how well the problem is explained. Some issues are clearly defined, while others are so vague they're nearly impossible to understand\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Test relevance\"}),\" checks whether the tests are fair and adequate: do they include all valid solutions, or are they too specific or not specific enough?\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Difficulty\"}),\" estimates how long it would take a professional to solve the problem. This ranges from quick fixes (under 15 minutes) to complex tasks (over 4 hours).\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"Based on the scores, problematic samples are removed from the dataset. Coding experts also provide feedback on aspects that may have been overlooked and flag other significant issues they see in the samples. The final step is to scan for false negatives caused by test misalignment and unclear descriptions. As a result, the final dataset only includes samples that meet a minimum quality threshold.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Step 2. Reject low-quality samples and complete missing fields for non-rejected samples\"}),\".\"]}),/*#__PURE__*/e(\"p\",{children:\"After removing low-quality samples, the next step is to test solution patches from the remaining samples. Annotators create Dockerfiles to be used for setting up the environment corresponding to each issue. Then annotators apply solution patches to confirm they function correctly, eliminating the possibility of tests failing due to a setup error. Once the environments are configured and running, annotators run the unit tests to check if the proposed solution solves the problem.\"}),/*#__PURE__*/e(\"p\",{children:\"A sample is considered correct only if it passes both types of unit tests, which are:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"FAIL_TO_PASS: Designed to fail before the solution is implemented but passes afterward. Passing the FAIL_TO_PASS test confirms that the issue has been fixed.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"PASS_TO_PASS: These tests pass before and after applying the code changes, ensuring the proposed solution doesn't introduce any new bugs to the codebase.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"In summary, the final dataset consists of high-quality samples that successfully pass both the FAIL_TO_PASS and PASS_TO_PASS tests.\"}),/*#__PURE__*/e(\"h2\",{children:\"The outcome and benefits of dataset cleanup\"}),/*#__PURE__*/e(\"p\",{children:\"The resulting dataset has 5000 clean samples that are more suitable for assessing LLM coding skills than the original SWE benchmark. This thorough cleanup has several significant advantages that directly benefit model evaluation, particularly when testing advanced coding assistants like GitHub Copilot, Cursor AI, or open-source models available through Ollama.\"}),/*#__PURE__*/e(\"p\",{children:\"First, it allows for more realistic model evaluation through clearly defined tasks and relevant scenarios. The LLMs are tested in real-world conditions, leading to more accurate assessments of their abilities.\"}),/*#__PURE__*/e(\"p\",{children:\"This method introduces another key benefit: reproducibility. With standardized testing environments, results will be consistent no matter where or when we run the tests. This makes it easier to compare performance across different LLMs and monitor progress over time.\"}),/*#__PURE__*/e(\"p\",{children:\"All these adjustments contribute to increased credibility. Benchmarks that are user-friendly, conducive to realistic evaluations, and reproducible are inherently more trustworthy within the research community. The cleaned SWE benchmark is an invaluable tool for evaluating models in software engineering.\"}),/*#__PURE__*/e(\"h2\",{children:\"Further improvements and the prospect of a new benchmark\"}),/*#__PURE__*/e(\"p\",{children:\"Although cleaning up the dataset is a big step forward, there is still more to be done. At Toloka, we are leveraging our extensive network of coding experts to work on improving SWE-bench's usability and expand it into an entirely new benchmark. Let's look at some of the proposed adaptations.\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Add everyday coding tasks.\"}),\" The SWE-bench dataset doesn't represent the diverse real-world tasks enterprise developers face on a daily basis. Aiming to cover this gap, our research team is working on an extended version of SWE-bench that includes tasks from the enterprise world. After all, tasks should reflect developers\u2019 typical needs when they use GitHub Copilot, Cursor AI, Codeium or other popular coding assistants.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Include diverse prompts.\"}),\" SWE-bench mainly focuses on one type of prompt, restricting the evaluation of crucial programming skills. Adding a wider array of prompts, such as code translations, analytical tasks, and bare code generation, will expand the evaluated skill set and provide a more rounded view of LLMs' coding abilities.\\xa0\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Expand to include more programming languages\"}),\". Currently, there\u2019s SWE-bench for Python, SWE-bench Multimodal for JavaScript, and SWE-bench-java\u2026 well, for Java. Broadening the range to other programming languages like C# and TypeScript will create a versatile benchmark that supports testing on diverse use cases. Later, industry-specific languages like Fortran, COBOL, Lisp and C could be added.\\xa0\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Increase dataset size.\"}),\" The current SWE-bench has 21,527 samples, with only 5000 approved by human annotators. Augmenting the dataset with new data will make it more comprehensive.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Address the static nature of the dataset.\"}),\" Another concern is the dataset's static nature, which does not account for new coding practices and can quickly become outdated. The dataset will need to be continuously updated to ensure it stays relevant to modern developments in Python, JavaScript and Java.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Create more suitable data for assessing AI agents.\"}),\" Finally, creating new data designed explicitly for assessing AI agents will reflect the reality of modern AI development. This includes implementing feedback loops in which agents use output logs and error messages to refine their solutions, as well as capturing the agents' reasoning processes similar to the chain-of-thought approach used by \",/*#__PURE__*/e(n,{href:\"https://nebius.com/blog/posts/scaling-data-collection-for-training-swe-agents\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Nebius\"})}),\".\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"Why it matters\"}),/*#__PURE__*/e(\"p\",{children:\"Refined benchmarks offer a clearer, more accurate picture of how software development tools and models perform. By making evaluations more accessible and reliable, they empower researchers and developers to make well-informed decisions. Our work enhances the quality of benchmarking data, ensuring stronger, more meaningful assessments of proprietary and open-source models, whether you\u2019re considering the DeepSeek model family, Anthropic\u2019s Claude, Google\u2019s Gemini, or even Alibaba\u2019s Qwen.\"}),/*#__PURE__*/t(\"p\",{children:[\"At \",/*#__PURE__*/e(\"strong\",{children:\"Toloka\"}),\", our deep expertise in data annotation and benchmarking allows us to extend these improvements beyond SWE-bench. We look forward to opportunities to refine other benchmarks as we deliver high-quality, trustworthy data across a broad range of use cases and industry domains.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Ready to take your LLM's coding skills to the next level?\"}),/*#__PURE__*/t(\"p\",{children:[\"Improve your model assessments with our specialized coding data solutions. \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/coding-data\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Learn more\"})}),\"\\xa0\"]})]});export const richText1=/*#__PURE__*/t(a.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"More and more businesses are using AI agents to improve efficiency. In 2023, the AI agents market was worth \",/*#__PURE__*/e(n,{href:\"https://www.grandviewresearch.com/industry-analysis/ai-agents-market-report\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"$3.86 billion\"})}),\", and it is expected to grow quickly\u2014by about 45.1% every year from 2024 to 2030. This growth is happening because companies want more intelligent automation, AI improves natural language processing (NLP), and customers expect more personalized experiences. As AI agents continue to develop, they are becoming a key part of many business processes, making work faster and easier while reducing the need for human effort.\"]}),/*#__PURE__*/e(\"h2\",{children:\"The role of AI agents in AI automation\"}),/*#__PURE__*/e(\"p\",{children:\"AI automation uses artificial intelligence to handle repetitive tasks and processes without human intervention. In 2025 AI automation relies more and more on AI agents. These agents act as the brains behind automation. Automation that is not powered by AI is just a collection of pre-programmed actions. However, when AI agents are incorporated, workflows are optimized more efficiently through the agents' ability to think, adjust, and take relevant actions.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"A simple automated customer service system might answer basic questions but breaks down when a customer asks something outside the script. An AI agent, however, can understand the request, evaluate the situation, and either provide a response or pass it on to a human when necessary.\"}),/*#__PURE__*/e(\"h2\",{children:\"Understanding AI agents\"}),/*#__PURE__*/e(\"p\",{children:\"AI agents are intelligent systems that can perceive their environment, process information, and take action autonomously. They can be rule-based or utilize advanced machine learning algorithms to adapt and improve over time. These agents operate in different forms, including virtual assistants, chatbots, robotic process automation (RPA) bots, and autonomous systems.\"}),/*#__PURE__*/e(\"p\",{children:\"Autonomous AI agents are sophisticated systems that integrate two fundamental components to perform complex tasks:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Large Language Models (LLMs):\"}),\" These models excel at understanding and generating human-like text, which helps the agent to comprehend instructions, process information, and engage in natural language interactions. LLMs simulate human-level understanding and reasoning by recognizing patterns in massive datasets. So the agent can engage in conversations, interpret user queries, and synthesize information.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Decision-making and Action Models:\"}),\" These include reinforcement learning, rule-based logic, or specialized AI systems designed to interpret information and take actions beyond language processing. By combining LLMs with action-oriented models, autonomous agents can engage in conversations, interpret user queries, and execute tasks dynamically.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"This integration creates a powerful combination for autonomous AI agents. While LLMs provide the cognitive capabilities to process information and understand language, action-oriented models give the AI the ability to take action in response to that information. The terms \u201CAI agents\u201D and \u201CLLM agents\u201D are more commonly used when discussing AI systems that automate tasks, but action-oriented models power these agents behind the scenes.\"}),/*#__PURE__*/e(\"h2\",{children:\"How AI agents work\"}),/*#__PURE__*/e(\"p\",{children:\"AI agents are designed to function autonomously. They process information, make decisions, and act on those decisions. At their core, they combine advanced models and technologies that allow them to perceive the environment, reason through complex tasks, and execute actions without constant human supervision. Here's how these agents perform their tasks and adapt to new situations.\\xa0\"}),/*#__PURE__*/e(\"h3\",{children:\"1. Perception: understanding the environment\"}),/*#__PURE__*/e(\"p\",{children:\"The first step for any AI agent is understanding the environment around it. This is known as perception, where the agent collects data through text, sensors, or digital inputs. The goal is to transform raw data into something the agent can comprehend and process.\"}),/*#__PURE__*/e(\"p\",{children:\"For instance, a chatbot AI agent might begin by receiving a message from a user. It doesn\u2019t just see the words; it must interpret the intent behind the message. Similarly, a robot designed to navigate an office space might use sensors to detect obstacles or recognize specific objects. By processing this information, the AI agent gets a clear picture of its environment, which is essential for making decisions.\"}),/*#__PURE__*/e(\"h3\",{children:\"2. Reasoning: deciding on the best action\"}),/*#__PURE__*/e(\"p\",{children:\"Once the AI agent understands its environment, the next step is reasoning \u2013 figuring out what it should do with the information. The agent uses various machine learning algorithms to evaluate different possibilities and outcomes.\"}),/*#__PURE__*/e(\"p\",{children:\"Virtual assistant powered by AI agent and you ask to schedule a meeting, it doesn\u2019t just pick a random time. It considers your availability, preferences, the urgency of the meeting, and possibly even your calendar history. In other words, the AI agent thinks through the options and chooses the one that makes the most sense based on the situation at hand.\"}),/*#__PURE__*/e(\"h3\",{children:\"3. Action: implementing the decision\"}),/*#__PURE__*/e(\"p\",{children:\"The AI agent takes that step after deciding on the best course of action. This is where the action part of the cycle happens. Depending on its capabilities, the agent could take various actions, such as sending an email, recommending, or physically moving (if incorporated into an autonomic vehicle). For a customer service bot, this might mean offering a solution to a customer\u2019s issue. For a self-driving car, it could involve adjusting the steering to avoid an obstacle.\"}),/*#__PURE__*/e(\"p\",{children:\"The key here is that the action is not random; it\u2019s the result of careful processing and reasoning. The AI agent follows through with what it has determined is the most appropriate response.\"}),/*#__PURE__*/e(\"h3\",{children:\"4. Learning: evolving and improving over time\"}),/*#__PURE__*/e(\"p\",{children:\"One of the most powerful aspects of AI agents is their ability to learn and improve. After taking action, AI agents assess the outcome to determine whether it was successful. If the agent's action produces a positive result, it may reinforce the decision-making process. If not, the agent evaluates the mistake and adjusts its future behavior.\"}),/*#__PURE__*/e(\"p\",{children:\"In some cases, AI agents use reinforcement learning, where they\u2019re \u201Crewarded\u201D for actions that lead to positive outcomes and \u201Cpenalized\u201D for those that don\u2019t. Over time, this feedback helps the agent refine its decision-making process, much like how humans learn from experience. This ability to learn from past actions allows AI agents to handle more complex tasks and become more effective as they continually adjust based on new information and experiences.\"}),/*#__PURE__*/e(\"h3\",{children:\"5. Autonomy: operating independently\"}),/*#__PURE__*/e(\"p\",{children:\"At the heart of an AI agent is its autonomy. While some agents require human input for guidance, many can operate independently once trained. The combination of perception, reasoning, action, and learning enables them to carry out tasks without constant human oversight.\"}),/*#__PURE__*/e(\"p\",{children:\"For instance, in e-commerce, an AI agent might autonomously suggest products to customers based on their browsing history. It doesn\u2019t need to be told what to indicate every time\u2014it uses its knowledge and algorithms to make recommendations that it believes will match the customer\u2019s needs. Similarly, robots can autonomously monitor and maintain equipment in industrial settings, detecting issues before they become significant problems.\\xa0\"}),/*#__PURE__*/e(\"h3\",{children:\"6. Adaptability: responding to new challenges\"}),/*#__PURE__*/e(\"p\",{children:\"AI agents are also highly adaptable, meaning they can respond to new situations without explicit programming for each scenario. For instance, a home assistant can recognize a new voice or adapt to changes in a user\u2019s schedule. As they interact with users or navigate environments, they continuously refine their behavior and actions to suit the new information they gather.\"}),/*#__PURE__*/e(\"p\",{children:\"This adaptability makes AI agents incredibly useful across diverse industries\u2014from customer service to healthcare, logistics, and more. The more they experience, the better they become at undestanding needs and responding in ways that seem intuitive to humans.\"}),/*#__PURE__*/e(\"h2\",{children:\"How intelligent agents are trained\"}),/*#__PURE__*/e(\"p\",{children:\"AI agents are trained through data input, machine learning techniques, and iterative improvement. Training an AI agent involves teaching it to understand inputs, make decisions, and take action effectively. Here's how the process generally works:\"}),/*#__PURE__*/e(\"h3\",{children:\"1. Data collection and preparation\"}),/*#__PURE__*/e(\"p\",{children:\"Training starts with data. The more relevant and diverse data an AI agent receives, the better it can perform. This data might include text, images, sensor readings, or other environmental inputs. For example, if we\u2019re training a virtual assistant, we\u2019ll collect large amounts of conversational data, including user queries, feedback, and various interactions.\"}),/*#__PURE__*/e(\"p\",{children:\"Before the data can be used, it must be cleaned and processed. This might involve removing irrelevant or noisy information, normalizing data, and organizing it into a format the model can use. For instance, in training a language model, the text data might be tokenized (broken into smaller pieces like words or characters) so that the model can analyze it more effectively.\"}),/*#__PURE__*/e(\"h3\",{children:\"2. Supervised learning: teaching the agent what to do\"}),/*#__PURE__*/e(\"p\",{children:\"One of the most common methods for training AI agents is supervised learning, where the agent is given examples of inputs and the correct output for each. During this phase, the agent learns by matching inputs (e.g., a user\u2019s question) to the expected output (e.g., the correct response). Here\u2019s how it works:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"The AI is provided with labeled training data consisting of pairs of input-output examples.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"The agent tries to predict the output for each pair based on the input.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"When the agent\u2019s prediction is wrong, it receives feedback and adjusts its internal parameters to improve its next guess.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Over time, the agent becomes better at mapping inputs to correct outputs. In the case of a chatbot, this means the AI learns how to provide accurate responses to various user queries.\"}),/*#__PURE__*/e(\"h3\",{children:\"3. Reinforcement learning: learning through feedback\"}),/*#__PURE__*/e(\"p\",{children:\"Another powerful method used for training AI agents is reinforcement learning. In this type of training, an agent learns by interacting with its environment and receiving feedback through rewards or penalties. The process works like this:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"The agent takes an action based on its current understanding of the environment.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"If the action leads to a positive result (like completing a task), the agent receives a reward (often a numeric score).\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"The agent receives a penalty if the action results in a negative outcome (like an incorrect response).\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Through this process of trial and error, the agent learns to choose actions that maximize its rewards and minimize penalties. This is similar to how humans learn new behaviors, experimenting with different approaches until they find the most effective one. For example, a robot might initially take many wrong steps while navigating an unfamiliar room, but with each wrong turn, it learns which directions are more successful in avoiding obstacles.\"}),/*#__PURE__*/e(\"h3\",{children:\"4. Unsupervised learning: finding patterns in data\"}),/*#__PURE__*/e(\"p\",{children:\"In unsupervised learning, the agent is given data without explicit labels or instructions. Instead, the agent is tasked with independently finding patterns or structures within the data. This can be particularly useful when there\u2019s no pre-labeled data available.\"}),/*#__PURE__*/e(\"p\",{children:\"For example, an AI agent might analyze user behavior in a recommendation system without explicitly being trained on customer preferences data. The agent can make intelligent recommendations by identifying patterns in the data\u2014such as which products are often bought together or which movies similar users have watched.\"}),/*#__PURE__*/e(\"h3\",{children:\"5. Transfer learning: building on pre-existing knowledge\"}),/*#__PURE__*/e(\"p\",{children:\"AI agents can also benefit from transfer learning, which involves taking a pre-trained model and adapting it to new but related tasks. Instead of starting from scratch, the agent uses a model that has already learned something from another domain, like general language patterns or object recognition, and then fine-tunes it on the new task.\"}),/*#__PURE__*/e(\"p\",{children:\"This allows the agent to learn much faster and with fewer data points, as it can apply general knowledge from one area to another. For instance, an AI agent trained in general language understanding can be further trained to handle specific customer support queries more efficiently.\"}),/*#__PURE__*/e(\"h3\",{children:\"6. Fine-tuning and evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"After the agent has been trained, it undergoes a fine-tuning process. This involves tweaking the model to improve accuracy and efficiency based on specific performance metrics, such as response time, success rate, or user satisfaction. Fine-tuning might also involve running simulations or stress tests to identify areas for improvement. Once fine-tuned, the agent is evaluated through testing and real-world use. It is constantly monitored, and if necessary, it will be retrained with updated data to maintain or enhance its performance.\"}),/*#__PURE__*/e(\"h2\",{children:\"Types of AI agents\"}),/*#__PURE__*/e(\"p\",{children:\"AI agents come in different types, depending on how smart they are and what they can do. Some follow simple rules, while others can learn, make decisions, and even act without constant human supervision.\"}),/*#__PURE__*/e(\"h3\",{children:\"Simple reflex agents\"}),/*#__PURE__*/e(\"p\",{children:\"These simplest AI agents operate purely based on what they understand at the moment. They don't remember past events or anticipate future outcomes. They simply follow a set of predefined rules that tell them how to respond to specific inputs. Since they don't have memory, every situation is treated as if it's happening for the first time.\"}),/*#__PURE__*/e(\"h3\",{children:\"Model-based reflex agents\"}),/*#__PURE__*/e(\"p\",{children:\"Unlike simple reflex agents, these AI systems have an internal model of the world. They use this model to track changes and fill in missing information. This knowledge of the surroundings allows them to make better decisions even when they don\u2019t have a complete picture of what\u2019s happening.\"}),/*#__PURE__*/e(\"h3\",{children:\"Goal-based agents\"}),/*#__PURE__*/e(\"p\",{children:\"These agents don't just react to their environment; they actively pursue specific goals. They evaluate different actions, predict their outcomes, and choose the best path forward. This makes them ideal for situations where multiple steps are required to achieve a desired outcome, like a self-driving car that plans an optimal route to a destination. They work with a clearly defined goal state, a planning system to determine the best course of action, and a decision-making process that evaluates which steps bring them closer to success.\"}),/*#__PURE__*/e(\"h3\",{children:\"Learning agents\"}),/*#__PURE__*/e(\"p\",{children:\"Unlike static AI systems learning agents evolve. They analyze past experiences, receive feedback, and adjust their strategies to improve performance. Their structure includes a performance element that makes decisions, a critic that evaluates results, a learning module that refines strategies, and a problem generator that suggests new actions to explore for better outcomes.\"}),/*#__PURE__*/e(\"h3\",{children:\"Utility-based agents\"}),/*#__PURE__*/e(\"p\",{children:\"These agents don't just follow goals but weigh different options and choose the one that offers the best overall outcome. They assign values to various possibilities and aim to maximize their utility. They rely on a utility function that ranks different outcomes, a decision-making system that selects the most beneficial option, and a predictive model that estimates future consequences.\"}),/*#__PURE__*/e(\"h3\",{children:\"Hierarchical agents\"}),/*#__PURE__*/e(\"p\",{children:\"Hierarchical agents operate with a structured command system where higher-level agents set objectives, and lower-level agents handle the details. This approach is helpful in complex environments like robotics or manufacturing, where different layers of AI need to coordinate actions.\"}),/*#__PURE__*/e(\"p\",{children:\"They include a task decomposition mechanism to break down objectives, a command hierarchy to organize decision-making, and coordination protocols to ensure different parts of the system work together seamlessly.\"}),/*#__PURE__*/e(\"h3\",{children:\"Multi-agent systems (MAS)\"}),/*#__PURE__*/e(\"p\",{children:\"Instead of a single AI system handling everything, multi-agent systems consist of multiple AI agents working together. They can sometimes cooperate or compete to achieve individual or collective goals. In a warehouse, for example, different robots might coordinate to transport goods efficiently. Some systems focus on collaboration, where agents share information, while others introduce competition. Their key components include communication protocols for interaction, resource management strategies, and coordination rules to prevent conflicts.\"}),/*#__PURE__*/e(\"h2\",{children:\"Role of the AI tutors in AI agents development\"}),/*#__PURE__*/e(\"p\",{children:\"AI agents are getting smarter, but they don\u2019t improve on their own. AI tutors, expert trainers who teach AI agents how to understand language better, make better decisions, and provide more accurate responses play a key role in shaping them.\"}),/*#__PURE__*/e(\"p\",{children:\"Here\u2019s how AI tutors contribute to AI agent development:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Training AI to understand and communicate better. \"}),\"AI tutors refine AI agents by providing high-quality training data, fact-checking responses, and improving language models. This helps AI agents generate more natural and useful interactions.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Making AI more Ethical & Reliable. \"}),\"AI models can sometimes produce biased or misleading content. AI tutors help correct these issues, ensuring that AI agents provide fair, unbiased, and culturally appropriate responses.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"Without human guidance, AI agents can face ethical issues. AI tutors ensure they sound natural, give accurate information, and don\u2019t spread harmful content.\"}),/*#__PURE__*/e(\"h2\",{children:\"The future of AI agents\"}),/*#__PURE__*/e(\"p\",{children:\"AI agents transform how we interact with technology, from simple reflex systems to complex learning agents that adapt and improve over time. As these systems become more sophisticated, they are taking on tasks that once required human effort.\"}),/*#__PURE__*/e(\"p\",{children:\"Companies are increasingly looking to deploy AI agents to automate customer support, manage workflows, and optimize business operations. AI agents free up human workers to focus on more strategic and creative work by masterfully handling repetitive tasks. The more these systems learn and evolve, the more seamlessly they integrate into our daily lives.\"}),/*#__PURE__*/e(\"p\",{children:\"AI agents aren\u2019t replacing human intelligence anytime soon. However, they are certainly becoming powerful assistants, helping us work faster, smarter, and with fewer routine burdens. And as artificial intelligence continues to advance, we can only expect these digital assistants to become even more capable, making our lives easier in ways we have yet to imagine.\"})]});export const richText2=/*#__PURE__*/t(a.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"As \",/*#__PURE__*/e(\"strong\",{children:\"LLMs (Large Language Models) \"}),\"move toward widespread adoption, we expect they will be integrated into many consumer products in 2025. Yet one question is paramount: \",/*#__PURE__*/e(\"strong\",{children:\"How do we measure their true capabilities?\"}),\" Evaluating these models isn\u2019t just about ranking performance\u2014it\u2019s about uncovering their strengths and limitations, comparing models to guide improvements, and ensuring they deliver reliable, trustworthy results in real-world scenarios.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"LLM evaluation remains a complex challenge\"}),\". In this article, we break down \",/*#__PURE__*/e(\"strong\",{children:\"current evaluation methods,\"}),\" discuss \",/*#__PURE__*/e(\"strong\",{children:\"when and how to apply different approaches\"}),\", and share \",/*#__PURE__*/e(\"strong\",{children:\"practical examples\"}),\" from our evaluation projects at Toloka, where we tackle these challenges daily to help build smarter AI.\\xa0\"]}),/*#__PURE__*/e(\"h2\",{children:\"Computational evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"The first type of evaluation that can be performed is computational evaluation, which originates from traditional machine-learning evaluation methods. It relies on a gold standard sample and a statistical scoring method to compare a model\u2019s output against this reference. The result is a single score that quantifies how closely the model's output aligns with the reference standard. In other words, each task has a correct answer and a score that shows if the model \u201Cgot it right\u201D.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Commonly used metrics include ROUGE, BLEU, BERTScore, BARTScore, and others. This approach is fast, scalable, and effective for simple tasks with clear reference outputs. However, computational metrics often have a weak correlation with human judgment when applied to complex, open-ended tasks. It\u2019s best to use this type of evaluation for simple tasks, and augment it with human evaluation or other automated rating approaches to provide quick sanity checks when tuning models.\"}),/*#__PURE__*/e(\"h2\",{children:\"Human evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"Human evaluation is the holy grail for evaluating LLMs, especially when the raters are trained specialists. The process works by prompting the model with a question, then taking the answer and asking a human expert to evaluate it based on pre-specified criteria such as correctness, fluency, clarity, mentioning certain concepts, and so on. Evaluation can be done pointwise (assessing a single response) or side-by-side (comparing responses from different models). This is the most reliable type of evaluation, and unlike computational evaluation, which only works for text output, human evaluation is suitable for all output modalities, including video and audio.\\xa0\"}),/*#__PURE__*/t(\"p\",{children:[\"The downside is that human evaluation can be expensive and time-consuming. For complex tasks, it can also be challenging to find skilled experts who are qualified to perform high-quality evaluations. At Toloka, we rely on the \",/*#__PURE__*/e(n,{href:\"https://mindrift.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Mindrift\"})}),\" platform to access an extensive network of domain experts and skilled annotators with specialized training and sophisticated quality control processes.\\xa0\\xa0\"]}),/*#__PURE__*/e(\"h2\",{children:\"Automated evaluation\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Automated evaluation is similar to the human evaluation described above, except that human judgment is replaced with a strong LLM model. The model scores the answer based on several criteria or performs a side-by-side comparison, just like a human expert would do. This method is suitable for complex, open-ended tasks, but it is less time-consuming than human evaluation and works well for iterative measurements once an evaluation dataset has been created. Its main weakness is that it depends on the judge model\u2019s biases, so the LLM used for evaluation must be chosen carefully. Interestingly, research has shown that LLMs that solve problems exceptionally well are not always the best at judging solutions. Judging capabilities should be evaluated separately before relying on an LLM for automated evaluation.\"}),/*#__PURE__*/e(\"h2\",{children:\"Building datasets for rubric-based auto evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"At Toloka, we support LLM producers with several types of evaluations, including automated evaluation, which is best suited for text and reasoning problems. In addition to the standard LLM-based evaluation, we implement a human-in-the-loop approach to ensure that LLM judgments align with human evaluations, providing a final score for prompts across different criteria.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"600\",src:\"https://framerusercontent.com/images/tBxZ7sxqXISqw8Y0PQomwRXUrI.png\",srcSet:\"https://framerusercontent.com/images/tBxZ7sxqXISqw8Y0PQomwRXUrI.png?scale-down-to=512 512w,https://framerusercontent.com/images/tBxZ7sxqXISqw8Y0PQomwRXUrI.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/tBxZ7sxqXISqw8Y0PQomwRXUrI.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/tBxZ7sxqXISqw8Y0PQomwRXUrI.png 2400w\",style:{aspectRatio:\"2400 / 1200\"},width:\"1200\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Visual: Prompts in the dataset are judged by the LLM and validated by human experts to get a final score\"})}),/*#__PURE__*/e(\"p\",{children:\"Building datasets for this approach is challenging, but it is highly effective for complex and niche-domain problems. To create a comprehensive dataset, we involve human domain experts in a four-step process:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Creating prompts\"}),\" \u2013 Experts write prompts relevant to specific niche domains.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Defining evaluation rubrics\"}),' \u2013 Experts establish criteria to assess responses. Some rubrics may apply to the entire dataset (e.g., language fluency), while others may be question-specific (e.g., \"Does the answer explain a specific concept?\").']})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Generating a gold-standard answe\"}),\"r \u2013 Experts create a reference answer to use for comparing model outputs.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Validating data points\"}),\" \u2013 Experts review the dataset to ensure that prompts, reference answers, and rubrics align. If they encounter issues, they revise and refine the dataset.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"We have used this evaluation approach to test models across various domains, such as Law, Acoustics, Engineering, Mathematics, Natural Sciences, and Medicine. To achieve the highest quality of evaluation data, we collaborate with highly qualified experts\u2014most of our experts have Master\u2019s or PhD degrees and industry experience in relevant fields.\"}),/*#__PURE__*/e(\"h2\",{children:\"Human evaluation for video and audio domains\"}),/*#__PURE__*/e(\"p\",{children:\"For creative fields, automatic evaluation is still not effective. We highly recommend human evaluation for video and audio content, and the evaluation process needs to be precisely designed for the specific modality and use case.\\xa0\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"600\",src:\"https://framerusercontent.com/images/zH5bc8fiycW5ohSWuiga6Jj4hc.png\",srcSet:\"https://framerusercontent.com/images/zH5bc8fiycW5ohSWuiga6Jj4hc.png?scale-down-to=512 512w,https://framerusercontent.com/images/zH5bc8fiycW5ohSWuiga6Jj4hc.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/zH5bc8fiycW5ohSWuiga6Jj4hc.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/zH5bc8fiycW5ohSWuiga6Jj4hc.png 2400w\",style:{aspectRatio:\"2400 / 1200\"},width:\"1200\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Visual: Aggregated results of human evaluation of a video.\"})}),/*#__PURE__*/e(\"p\",{children:\"To create datasets for creative fields, we developed a specialized toolkit in collaboration with experts from the film industry, including producers and graphic designers. This toolkit enables the rapid creation of complex prompts and their corresponding evaluation criteria. Experts can customize prompts by selecting factors such as prompt complexity, desired features (e.g., the main focus of the video, character traits like age or species, and background details), and camera settings (e.g., location, angle, and movements).\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"697\",src:\"https://framerusercontent.com/images/IvtIJXsLe4BfdZFoguJ47TShvsc.png\",srcSet:\"https://framerusercontent.com/images/IvtIJXsLe4BfdZFoguJ47TShvsc.png?scale-down-to=512 512w,https://framerusercontent.com/images/IvtIJXsLe4BfdZFoguJ47TShvsc.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/IvtIJXsLe4BfdZFoguJ47TShvsc.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/IvtIJXsLe4BfdZFoguJ47TShvsc.png 2400w\",style:{aspectRatio:\"2400 / 1394\"},width:\"1200\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Visual: Dataset Toolkit for evaluating creatives.\"})}),/*#__PURE__*/e(\"p\",{children:\"Our approach is a scalable and effective way to create evaluation datasets for creative models. When used in conjunction with human evaluation, it gives model producers a detailed picture of the strengths and weaknesses of their models.\"}),/*#__PURE__*/e(\"h2\",{children:\"Do you need an in-depth evaluation of your model?\"}),/*#__PURE__*/t(\"p\",{children:[\"Before pushing a model to market, you may want to run multiple types of evaluations to understand how it will perform in real-life scenarios. We can support you with \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/evaluation\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"any approach to evaluation\"})}),\", whether it\u2019s evaluating complex text and reasoning capabilities or creative outputs. Custom evaluation datasets can align model performance with your expectations for specific domains, skills, and scenarios.\\xa0\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://toloka.ai/talk-to-us\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Connect with our team\"})}),\" to get a tailored solution. \"]})]});export const richText3=/*#__PURE__*/t(a.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Toloka\u2019s mission is to empower businesses with high-quality data to develop safe, responsible and trustworthy AI products.\"}),/*#__PURE__*/e(\"p\",{children:\"In fact, we have championed Responsible AI across 10 years of company growth by focusing on data excellence, leading the way in social impact with global opportunities for data annotators, and contributing to the AI community with scientific research.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"We recognize our role in shaping the future of AI alongside our customers, and we uphold the highest standards of privacy, security, safety, and fairness in AI development.\\xa0\"}),/*#__PURE__*/e(\"h2\",{children:\"Safety and evaluations: Ensuring AI integrity\"}),/*#__PURE__*/e(\"p\",{children:\"Above all else, we support our customers in meeting the highest safety and performance standards for their AI systems. Our data and evaluation services go beyond standard assessments, providing a framework for enhancing AI reliability and performance.\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Our comprehensive risk assessments and safety evaluations are customized to the model\u2019s use case\u2014coding, visual models, reasoning, or industry-specific needs.\\xa0\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"In rigorous testing and evaluations, we focus on real-world scenarios to ensure AI systems are ready for deployment.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"We collaborate with partners to develop robust safety benchmarks and domain-specific benchmarking tailored to specialized use cases. We are continuously raising the bar of industry standards on safety and reliability.\\xa0\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"AI Agents benefit from our pioneering safety evaluations as we advance best practices through red teaming, benchmarking, and other techniques for agentic safety.\"})})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://toloka.ai/evaluation\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Learn more\"})}),\" about our approach to evaluation and safety.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Community impact: Fair treatment of the humans behind AI\"}),/*#__PURE__*/e(\"p\",{children:\"Human insight remains essential in AI development. Even with advances in self-supervised learning, human expertise is critical for evaluation pipelines and domain-specific datasets.\"}),/*#__PURE__*/e(\"p\",{children:\"We built the Mindrift platform as a global community for data production where domain experts contribute to AI projects focused on their areas of expertise. We prioritize the well-being of our experts, annotators, and contributors by guaranteeing fair hourly rates, flexible working conditions, and ethical task design.\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Contributors around the world find opportunities for flexible part-time engagement, with fair and transparent compensation.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Ethical task design ensures that AI annotators and experts work in environments that prioritize safety and respect. We verify that contributors meet legal age requirements and limit their exposure to explicit or potentially toxic content.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Contributors work within structured task pools that match their expertise, and they can opt out of tasks without penalty.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Open communication means contributors have access to real-time communication channels for collaboration on tasks, and they can always provide feedback to improve our processes and workflows.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Transparency is paramount. We are open about our processes and people, ensuring fair treatment of all contributors.\\xa0\"}),/*#__PURE__*/t(\"p\",{children:[\"Learn more about the \",/*#__PURE__*/e(n,{href:\"https://mindrift.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Mindrift\"})}),\" platform.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Data excellence: Diversity and reliability\"}),/*#__PURE__*/e(\"p\",{children:\"We know that our data is behind powerful technology, and we take that responsibility very seriously. By delivering high-quality data that is accurate, truthful, and free of biases, we help AI applications remain grounded in real-world facts and perform effectively in diverse scenarios.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Here are some of the ways we support data quality in data production pipelines:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Our global community of domain experts and contributors are vetted and tested to verify their qualifications. Then they complete\\xa0 targeted onboarding specific to each project they participate in.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Quality Assurance specialists verify tasks in tailored quality workflows that are designed to meet the unique requirements of each project.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Platform-wide anti-fraud measures protect data integrity:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Our algorithms leverage behavioral signals and predictive methods to accurately detect and mitigate fraudulent activity on the data production platform.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Automated anomaly detection catches problematic behavior among contributors.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Our anti-fraud team investigates flagged activity case by case, and the results of every investigation are fed into algorithms to continuously improve fraud protection.\"})})]}),/*#__PURE__*/t(\"p\",{children:[\"Discover how we deploy high-quality data for real-world applications in our \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/success-stories\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Success Stories\"})}),\".\"]}),/*#__PURE__*/e(\"h2\",{children:\"Scientific research: Driving the AI industry forward\"}),/*#__PURE__*/e(\"p\",{children:\"Technological innovation at Toloka is rooted in reproducible research with a commitment to open inquiry, intellectual rigor, integrity, and collaboration. Our research team pushes our services to the forefront of the industry with pioneering approaches to data collection, model evaluation, and fine-tuning.\"}),/*#__PURE__*/e(\"p\",{children:\"Scientific research has been at the core of our identity since Toloka was founded in 2014. To promote responsible AI development through collaboration, we prioritize the following initiatives:\\xa0\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Our research fellowship program supports global AI researchers.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"We contribute to the AI ecosystem by sharing high-quality datasets and post-training methods with the research and open-source community.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Our research papers are published at top AI conferences (NeurIPS, ICML, CVPR, NAACL, AAAI, and others).\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"We collaborate with leading institutions like Hugging Face, Meta AI, Microsoft, Cohere for AI, MIT, Penn State, University of Oslo, and the University of Amsterdam.\"})})]}),/*#__PURE__*/t(\"p\",{children:[\"We also share insights through public media and educational webinars. Explore our latest \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/research\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"research initiatives\"})}),\".\"]}),/*#__PURE__*/e(\"h2\",{children:\"Privacy and security: A foundation of trust\"}),/*#__PURE__*/e(\"p\",{children:\"Responsible data production starts with an unwavering commitment to data privacy and security. We integrate privacy principles at every stage of our processes, making data protection a core consideration from the outset.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Key elements of our approach include:\\xa0\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Our dedicated security and privacy team continuously monitors, assesses, and enhances security measures across all systems and workflows.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"We employ a robust security framework that includes strict access controls, data encryption, and regular security assessments.\\xa0\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"To minimize risks to personal data, we prioritize the use of de-identified and synthetic data for AI training whenever possible.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"We ensure that all third-party partners for AI systems and AI tools or add-ons comply with our stringent privacy and security standards through rigorous vendor assessments.\"})})]}),/*#__PURE__*/t(\"p\",{children:[\"Learn more on our \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/security\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Security and Privacy portal\"})}),\".\"]}),/*#__PURE__*/e(\"h2\",{children:\"Shaping a responsible AI future\"}),/*#__PURE__*/e(\"p\",{children:\"At Toloka, our approach to responsible AI is built on trust, security, excellence, and fairness.\\xa0 By adhering to responsible practices, we are driving AI forward in a way that benefits businesses and society as a whole.\"}),/*#__PURE__*/e(\"p\",{children:\"Together, we can create a future where integrity is at the core of technology.\"})]});export const richText4=/*#__PURE__*/t(a.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"At Toloka, data security and privacy are at the core of everything we do. As part of our ongoing commitment to safeguarding the data of our users and clients, we\u2019re excited to introduce our Bug Bounty Program\u2014an initiative that invites security researchers and ethical hackers to help us identify and address vulnerabilities before they can be exploited.\"}),/*#__PURE__*/e(\"h2\",{children:\"What is the Toloka Bug Bounty Program?\"}),/*#__PURE__*/e(\"p\",{children:\"Our bug bounty program is designed to enhance the security of our expert platform, Mindrift Web and Mobile versions by leveraging the expertise of the global security research community. By participating, researchers can help us fortify our defenses while earning rewards for their contributions.\"}),/*#__PURE__*/e(\"h2\",{children:\"Scope and Eligibility\"}),/*#__PURE__*/e(\"p\",{children:\"The program focuses on identifying critical security vulnerabilities while excluding lower-risk issues such as denial-of-service attacks, non-sensitive information disclosure, and minor misconfigurations. We welcome submissions that demonstrate real security risks, including authorization bypasses, account takeover and business logic flaws.\"}),/*#__PURE__*/e(\"p\",{children:\"Researchers worldwide can participate, except those from restricted regions as outlined in our guidelines. We follow HackerOne\u2019s Gold Standard Safe Harbor policy, ensuring that ethical hacking within our program remains legally protected and aligned with responsible disclosure best practices.\"}),/*#__PURE__*/e(\"h2\",{children:\"Bounty Rewards and Submission Process\"}),/*#__PURE__*/e(\"p\",{children:\"We offer competitive rewards of up to $3,000 USD based on the severity of discovered vulnerabilities, categorized using the Common Vulnerability Scoring System (CVSS).\"}),/*#__PURE__*/e(\"p\",{children:\"Valid reports must include a proof of concept, impact assessment, and detailed reproduction steps. Submissions should be made via our HackerOne platform, where researchers will receive responses within three days, with a full triage process completed within seven days. Bounties will be issued within 30 days of a verified vulnerability.\"}),/*#__PURE__*/e(\"h2\",{children:\"Our Commitment to Security\"}),/*#__PURE__*/e(\"p\",{children:\"Launching a public bug bounty program reflects the maturity of our security practices. Unlike companies that fear the costs of fixing vulnerabilities, we believe in proactive security and addressing issues before they reach production. Our internal SLA mandates fixing critical vulnerabilities within five days and all other vulnerabilities within 30 days.\"}),/*#__PURE__*/e(\"h2\",{children:\"Join the Program\"}),/*#__PURE__*/t(\"p\",{children:[\"We invite security researchers to help us strengthen our defenses while earning rewards for their expertise. To get started, visit our \",/*#__PURE__*/e(n,{href:\"https://hackerone.com/toloka_ai\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"HackerOne program page\"})}),\". For any inquiries, reach out to us at security@toloka.ai.\"]}),/*#__PURE__*/e(\"p\",{children:\"Together, we can make Toloka\u2019s ecosystem more secure for everyone.\"})]});export const richText5=/*#__PURE__*/t(a.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"What sets a reasoning system apart, how to test for it, and how to achieve it\"})}),/*#__PURE__*/e(\"h2\",{children:\"New benchmarking results compare the\\xa0latest models\"}),/*#__PURE__*/t(\"p\",{children:[\"A few weeks ago, DeepSeek made waves with their \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/abs/2501.12948\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"latest R1 model\"})}),\", pushing ChatGPT out of the #1 spot in the Apple App Store and prompting tech stock selloffs that sent Nvidia share prices plummeting 17%.\"]}),/*#__PURE__*/t(\"p\",{children:[\"The claim is that DeepSeek R1 \u2014 an open-weights [1] model produced by a small Chinese tech lab \u2014 rivals OpenAI\u2019s o1 model in performance, but at a fraction of the cost for training and inference. The o1 model represents a shift towards \",/*#__PURE__*/e(\"strong\",{children:\"adaptive reasoning systems\"}),\" that dynamically scale their computation via test-time scaling, where models allocate more computational resources as needed during inference. Reasoning models have more powerful problem-solving capabilities than traditional LLMs, which have less capacity for adaptive allocation of compute. To access these capabilities in their latest o1 model, OpenAI charges six times more than for GPT-4o. DeepSeek offers the same type of reasoning system, but their model is open source.\"]}),/*#__PURE__*/e(\"p\",{children:\"Setting aside the geopolitical aspect, open vs closed source questions, and implications of an imminent decline in training and inference costs, let's focus on DeepSeek's biggest splash \u2014 the performance parity.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"The Toloka Research team has been testing R1 and o1 and analyzing existing evaluations, and our findings bring R1\u2019s dominance into question.\"})}),/*#__PURE__*/e(\"h4\",{children:\"Read on to learn:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"How R1 really compares to o1\"})})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Where we\u2019ve exposed the most significant gaps\"})})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"What makes o1 stand apart from all the current reasoner models\"})})}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"How to approach closing the gap\"})}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]})]}),/*#__PURE__*/e(\"h2\",{children:\"Public comparisons show R1\\xa0on\\xa0par\\xa0with\\xa0o1\"}),/*#__PURE__*/e(\"p\",{children:\"R1 is commonly quoted to be on par with o1 by AI enthusiasts and the media, based on public benchmarks that show remarkably similar performance. Let's look at available comparisons of R1 and o1. \"}),/*#__PURE__*/e(\"h4\",{children:\"DeepSeek report\"}),/*#__PURE__*/e(\"p\",{children:\"This comparison table from DeepSeek\u2019s own report shows R1 and o1 neck and neck on major benchmarks. The majority of articles comparing R1 to o1 are based on this table, or tiny-scale manual vibe checks using a handful of prompts.\"}),/*#__PURE__*/e(\"figure\",{className:\"framer-table-wrapper\",children:/*#__PURE__*/e(\"table\",{children:/*#__PURE__*/t(\"tbody\",{children:[/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Domain\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Benchmark\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"o1\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"R1\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"Mathematics\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"MATH-500 (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"96.4%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"97.3%\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"AIME 2024 (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"79.2%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"79.8%\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"Coding\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"Codeforces (Percentile)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"96.6%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"96.3%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"LiveCodeBench (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"63.4%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"65.9%\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"SWE-bench Verified (Resolved)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"48.9%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"49.2%\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"General\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"DROP (3-shot F1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"90.2%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"92.2%\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"MMLU (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"91.8%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"90.8%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"GPQA Diamond (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"75.7%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"71.5%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"SimpleQA (Correct)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"47.0%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"30.1%\"})})]})]})})}),/*#__PURE__*/e(\"h4\",{children:\"Popular online leaderboards\"}),/*#__PURE__*/t(\"p\",{children:[\"Additional \u201Cgo-to\u201D evaluation sources are the full \",/*#__PURE__*/e(n,{href:\"https://livebench.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"LiveBench\"})}),\" leaderboard and ChatBot Arena [2].\"]}),/*#__PURE__*/e(\"figure\",{className:\"framer-table-wrapper\",children:/*#__PURE__*/e(\"table\",{children:/*#__PURE__*/t(\"tbody\",{children:[/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Domain\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Benchmark\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"o1\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"R1\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"General\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"ChatBot Arena (Rating)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"1352 (+6/-7)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"1361 (+7/-8)\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"Data Science\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"LiveBench (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"65.5%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"69.8%\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"Instruction Following\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"LiveBench (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"81.6%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"80.5%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"Reasoning (Spatial & Logic\\xa0Puzzles)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"LiveBench (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"91.6%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"83.2%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"Language (Grammar & Comprehension)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"LiveBench (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"65.4%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"48.5%\"})})]})]})})}),/*#__PURE__*/e(\"h4\",{children:\"Style comparisons\"}),/*#__PURE__*/t(\"p\",{children:[\"There are also numerous anecdotal reports on R1\u2019s superior writing style, generally described as \u201Cmore creative and fun, if a bit unhinged,\u201D which is in line with some \",/*#__PURE__*/e(n,{href:\"https://github.com/lechmazur/writing/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"independent\"})}),\" \",/*#__PURE__*/e(n,{href:\"https://github.com/EQ-bench/EQ-Bench\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"evaluations\"})}),\".\"]}),/*#__PURE__*/e(\"h4\",{children:\"General sentiment\"}),/*#__PURE__*/e(\"p\",{children:\"The widespread narrative based on those evaluations is that the models are more or less equal. R1 seems to be slightly better at math and coding and have a more free and creative writing style, while o1 is ostensibly somewhat better at factuality, question answering, and instruction following, with a writing style that focuses on careful structure, grammar, and logic.\"}),/*#__PURE__*/e(\"p\",{children:'All of these differences can be vaguely attributed to the focus: R1 training was \"less restricted\", with an emphasis on math and coding domains that lend themselves well to reinforcement learning, while o1 was presumably \"more guided\", with more attention to world knowledge and alignment. OpenAI invested heavily in o1\\'s alignment as a public-facing product via ChatGPT, ensuring the model\\'s safety and adaptation to consumer preferences across general domains. It all seems to add up.'}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"But we think there\u2019s more to the story.\"})}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"h2\",{children:\"Long-tail benchmarks reveal gaps\"}),/*#__PURE__*/e(\"p\",{children:\"As soon as we leave the beaten path, alternative benchmarks paint a different picture. The Toloka Research team investigated evaluations in niche subdomains and uncommon domains and noted quantitative and qualitative gaps in model performance.\"}),/*#__PURE__*/e(\"h3\",{children:\"Niche subdomains\"}),/*#__PURE__*/t(\"p\",{children:[\"A few months ago, Toloka released \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/math-benchmark\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"U-MATH\"})}),\" \u2014 a benchmark to test LLMs on university-level mathematics, sourced from unpublished problems used in the curriculum at top US universities. We developed the benchmark precisely to represent an uncommon, under-explored niche within the math domain \u2014 real-world university math. According to our U-MATH evaluation, \",/*#__PURE__*/e(\"strong\",{children:\"R1 does not seem to have an edge over o1 in mathematical reasoning.\"}),\" [3][4][5]\"]}),/*#__PURE__*/e(\"figure\",{className:\"framer-table-wrapper\",children:/*#__PURE__*/e(\"table\",{children:/*#__PURE__*/t(\"tbody\",{children:[/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Benchmark\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"o1\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"R1\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"MATH-500 (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"96.4%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"97.3%\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"AIME 2024 (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"79.2%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"79.8%\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"U-MATH (TextHard, Accuracy)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"90.5%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"88.2%\"})})]})]})})}),/*#__PURE__*/t(\"p\",{children:[\"We continued to study niche subdomains by examining coding, the other domain where R1 is considered to have an edge over o1. In terms of coding benchmarks, \",/*#__PURE__*/e(n,{href:\"https://aider.chat/2024/12/21/polyglot.html#the-polyglot-benchmark\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Aider\u2019s Polyglot\"})}),\" is a good niche example, focusing on code editing tasks in a large number of different programming languages, in contrast to the more widespread from-scratch Python generation. Aider's evaluations \",/*#__PURE__*/e(\"strong\",{children:\"contradict the story of R1 beating o1 in coding\"}),\", similar to our conclusions on U-MATH.\"]}),/*#__PURE__*/e(\"figure\",{className:\"framer-table-wrapper\",children:/*#__PURE__*/e(\"table\",{children:/*#__PURE__*/t(\"tbody\",{children:[/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Benchmark\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"o1\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"R1\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"Codeforces (Percentile)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"96.6%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"96.3%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"LiveCodeBench (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"63.4%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"65.9%\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"SWE-bench Verified (Resolved)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"48.9%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"49.2%\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Aider Polyglot (Correct)\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"61.7%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"56.9%\"})})]})]})})}),/*#__PURE__*/e(\"h3\",{children:\"Unusual domains\"}),/*#__PURE__*/e(\"p\",{children:\"Conitnuing in this vein, we've investigated benchmarks that assess altogether less conventionally tested domains and capabilities, not just less conventional subdomains. There, a pronounced gap in favor of o1 is observed\"}),/*#__PURE__*/e(\"figure\",{className:\"framer-table-wrapper\",children:/*#__PURE__*/e(\"table\",{children:/*#__PURE__*/t(\"tbody\",{children:[/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Benchmark\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"o1\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"o1-mini\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"R1\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"\u03BC-MATH (F1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"90.1%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"83.4%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"84.3%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"ARC-AGI-1 (Accuracy)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"32.0%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"7.8%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"15.8%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"LLM Chess (Wins)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"57.4%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"30.0%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"22.6%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"NPR Sunday Puzzle (Correct)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"59.0%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"26.0%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"35.0%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"NYT Connections Extended\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"70.8%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"27.0%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"38.6%\"})})]})]})})}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://toloka.ai/blog/math-benchmark/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"\u03BC-MATH\"})}),\" is our other benchmark on university mathematics, based on U-MATH but intended for meta-evaluations \u2014 i.e., assessing the performance of LLMs as automatic solution evaluators instead of problem solvers. Judging solutions by testing them against a golden label proves to be a challenging skill in its own right\",/*#__PURE__*/e(\"em\",{children:/*#__PURE__*/e(\"strong\",{children:\" \"})}),\"[6], and it is a somewhat atypical skill to evaluate and train for. Here we observe R1\u2019s score closer to that of o1-mini, not o1. [7]\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://arcprize.org/blog/r1-zero-r1-results-analysis\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"ARC\"})}),\" tests AI systems on their ability to adapt to novel situations, showing R1 lagging well behind o1.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://maxim-saplin.github.io/llm_chess/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"LLM Chess Leaderboard\"})}),\" again places R1 behind o1 and closer to o1-mini. The evaluation also reports that R1 makes an average of 52.93 illegal moves per 1,000 attempts, compared to o1 and o1-mini with 0 and 4.29 illegal moves, respectively [8].\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2502.01584\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Northeastern University Programming Research Lab Study\"})}),' tested the models on the NPR Sunday Puzzle Challenge, which are difficult wordplay problems based on common knowledge. In addition to the large gap between R1 and o1, researchers also reported several \"failure modes\" that occur with R1 and never occur with o1 or o1-mini, such as R1 explicitly stating \"I give up\" 23.9% of the time or failing to stop the generation 8.5% percent of the time. In 6.6% of cases, R1 exhibtis erratic answer switches, which only happens in 0.5% of cases for o1-mini and does not happen for o1.']})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Similarly, as reported by the \",/*#__PURE__*/e(n,{href:\"https://github.com/lechmazur/nyt-connections\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"NYT connections leaderboard\"})}),\", R1 is far behind o1 in solving puzzles that involve finding semantic subgroups within word sets.\"]})})]}),/*#__PURE__*/e(\"h3\",{children:\"Superior generalization and reliability place o1 in a league of its own\"}),/*#__PURE__*/t(\"p\",{children:[\"The long-tail benchmarks above are pertinent because \",/*#__PURE__*/e(\"strong\",{children:\"they are unconventional, testing for novelty and robustness\"}),\". So here\u2019s our claim: \",/*#__PURE__*/e(\"strong\",{children:\"o1 has greater generalization and reliability than R1\"}),\". In terms of reliability in particular, we can see that both o1 and o1-mini are superior to R1.\"]}),/*#__PURE__*/e(\"p\",{children:\"As we continued investigating, we found that other publicly available evaluations support our conjecture.\"}),/*#__PURE__*/t(\"p\",{children:[\"From the standpoint of lesser generalization, we expect R1 not only to perform worse on niche subdomains or novel domains, but also to display degraded performance on new, unseen tasks for familiar suites. Our expectation is confirmed by \",/*#__PURE__*/e(n,{href:\"https://matharena.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"MathArena\"})}),\"\u2019s recent report on AIME 2025 results, highlighting performance on problems published after the release of o1 and R1 [9]. With the unseen tasks, performance drops significantly for R1 but not for o1.\"]}),/*#__PURE__*/e(\"figure\",{className:\"framer-table-wrapper\",children:/*#__PURE__*/e(\"table\",{children:/*#__PURE__*/t(\"tbody\",{children:[/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Benchmark\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"R1\"})})}),/*#__PURE__*/e(\"th\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"o1\"})})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"AIME 2024 (Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"79.8%\"})})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"79.2%\"})})]}),/*#__PURE__*/t(\"tr\",{children:[/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"AIME 2025 (avg I & II, Pass@1)\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:\"70.0%\"})}),/*#__PURE__*/e(\"td\",{children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"79.2%\"})})})]})]})})}),/*#__PURE__*/e(\"p\",{children:\"Reliability, in turn, plays a vital role in adversarial robustness and consistency, both lacking in R1 compared to o1 and o1-mini.\"}),/*#__PURE__*/e(\"ul\",{children:/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://blogs.cisco.com/security/evaluating-security-risk-in-deepseek-and-other-frontier-reasoning-models\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"CISCO report\"})}),\" a 100% success rate on their attempts to breach R1, compared to 26% for the o1-preview, and there are other \",/*#__PURE__*/e(n,{href:\"https://www.holisticai.com/blog/ensuring-ai-safety-holistic-ai-deepseek-audit\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"similar\"})}),\" \",/*#__PURE__*/e(n,{href:\"https://cdn.prod.website-files.com/6690a78074d86ca0ad978007/679bc2e71b48e423c0ff7e60_1%20RedTeaming_DeepSeek_Jan29_2025%20(1).pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"reports\"})}),\" on R1\u2019s safety limitations [10].\"]})})}),/*#__PURE__*/e(\"ul\",{children:/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"A recent \",/*#__PURE__*/e(n,{href:\"https://github.com/vectara/hallucination-leaderboard\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"update of Vectara's hallucination benchmark\"})}),\" shows R1's hallucination rate at 14.3%, while o1 and o1-mini only display 2.4% and 1.4%, respectively.\"]})})}),/*#__PURE__*/t(\"p\",{children:[\"Both proper generalization and reliability are extremely important for AI systems, often cited as the primary bottlenecks of agentic applications. An extension of our claim is that superior \",/*#__PURE__*/e(\"strong\",{children:\"generalization and reliability put o1 in a league of its own\"}),\" among all the currently available reasoning models. While we agree with the popular narrative that R1 has a different focus from o1, the o1 emphasis on alignment and broad domain coverage results in a qualitative difference that goes far beyond performance accents.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Coming up, we'll broaden the discussion of generalization and reliability in models and will then discuss \",/*#__PURE__*/e(\"strong\",{children:\"ways to enhance these key properties in a reasoning system\"}),\", so stay tuned.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"h2\",{children:\"Beyond o1 vs R1: studying trends and biases\"}),/*#__PURE__*/e(\"p\",{children:\"Bringing in more models and metrics can help us get a bigger-picture perspective on the performance patterns, instead of only focusing on o1-to-R1 comparisons from standalone benchmarks. Let's turn again to our own suite of U-MATH and \u03BC-MATH datasets.\"}),/*#__PURE__*/e(\"h3\",{children:\"Recognizing patterns\"}),/*#__PURE__*/t(\"p\",{children:[\"You can study the U-MATH and \u03BC-MATH leaderboard \",/*#__PURE__*/e(n,{href:\"https://huggingface.co/collections/toloka/u-math-and-math-university-level-math-evaluation-67515d973f63bfd451ebdc30\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"on Hugging Face\"})}),\". Here are the aggregate scores, ranked by \u03BC-MATH F1.\"]}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"524\",src:\"https://framerusercontent.com/images/whZ0dtbqB84lIpWvC9MlhCaHw.svg\",srcSet:\"https://framerusercontent.com/images/whZ0dtbqB84lIpWvC9MlhCaHw.svg?scale-down-to=512 512w,https://framerusercontent.com/images/whZ0dtbqB84lIpWvC9MlhCaHw.svg 856w\",style:{aspectRatio:\"1143 / 1048\"},width:\"571\"}),/*#__PURE__*/e(\"p\",{children:\"The graph reveals three major performance trends:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Judgment vs. problem-solving:\"}),' In non-reasoning models, judgment and problem-solving performance improve together \u2014 up to a point. Beyond that, they diverge, with better problem-solving correlating with weaker judgment. This puts some numbers behind our statement that judgment is a distinct skill and suggests an inherent tradeoff between the two, with models in a sense being \"forced to specialize\" in either one or the other.']})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Reasoners extend the Pareto frontier\"}),\" by breaking out of this tradeoff, demonstrating their superior generalization and advancing beyond the previous generation of models [11].\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"o1 stands apart\"}),\" by pushing a step beyond other reasoning models into its own category.\"]})})]}),/*#__PURE__*/e(\"h3\",{children:\"Investigating the tradeoffs\"}),/*#__PURE__*/e(\"p\",{children:\"We won\u2019t go into much detail here [12], but what we find is that the problem-solving vs judgment tradeoff mentioned above translates into behavioral differences among the models, yielding two distinctive judgment styles:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Lenient judges\"}),\": More verbose, inclined toward lengthy derivations and better at comparing complex expressions, but prone to losing track \u2014 leading to fewer false negatives but more false positives.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Conservative judges\"}),\": More structured and precise, but often anchored on the exact form of the reference answer or golden label \u2014 leading to fewer false positives but more false negatives.\"]})})]}),/*#__PURE__*/t(\"p\",{children:[\"Good judgment depends on balancing mathematical problem-solving with general structured logic and instruction-following, and an ideal model would have \",/*#__PURE__*/e(\"strong\",{children:\"strong domain-specific reasoning skills\"}),\" while maintaining \",/*#__PURE__*/e(\"strong\",{children:\"high reliability and coherence\"}),\" in order to successfully apply them.\"]}),/*#__PURE__*/e(\"p\",{children:\"To understand this balance better, we can decompose the F1 score into True Positive Rate (TPR), which measures correct positive classifications, and True Negative Rate (TNR), which tracks correct rejections of incorrect solutions. Higher TPR means fewer false negatives, while higher TNR means fewer false positives.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"330\",src:\"https://framerusercontent.com/images/drlc2iCzKMjLzayYiQ2JlT0OQCw.svg\",srcSet:\"https://framerusercontent.com/images/drlc2iCzKMjLzayYiQ2JlT0OQCw.svg?scale-down-to=512 512w,https://framerusercontent.com/images/drlc2iCzKMjLzayYiQ2JlT0OQCw.svg 1000w\",style:{aspectRatio:\"1334 / 661\"},width:\"667\"}),/*#__PURE__*/t(\"p\",{children:[\"We can observe all the same trends with this chart: non-reasoning systems exhibiting the performance tradeoff [13]\",/*#__PURE__*/e(\"em\",{children:\",\"}),\" reasoning systems pushing the frontier away [14], and o1 advancing a step further still. What we can also see is that on top of being the best-performing model, \",/*#__PURE__*/e(\"strong\",{children:\"o1 is among the most balanced ones\"}),\", and is the most balanced amid the tested reasoners [15].\"]}),/*#__PURE__*/e(\"p\",{children:\"Studying the previous generation of models informs us on how to approach this balancing with the next:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Balanced training makes for a more balanced model\"}),\". We can see in action how transitioning from math specialists to generalist models leads to better, more well-rounded judgment performance. A balanced training setup integrates the depth of formal reasoning with breadth of domain coverage and coherence.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Reducing capability amplifies training-induced biases\"}),\". Lenient models tend to become even more lenient when scaled down to smaller sizes, and conservative models become more conservative. A model needs to have appropriate capabilities that allow it to generalize over the things that you\u2019re balancing.\"]})})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"The key point is:\"}),\" Powerful reasoning systems have the appropriate capabilities and already excel at formal reasoning. Training focused on diversifying domains and improving coherence will help guiding them toward generalization and reliability.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"h2\",{children:\"Building more generalizable and reliable models: It\u2019s all in the data\"}),/*#__PURE__*/e(\"p\",{children:\"We\u2019ve discussed at length that a reasoning system would require generalization and reliability, demonstrating o1\u2019s excellence on these dimensions and its resulting superb performance.\"}),/*#__PURE__*/e(\"p\",{children:\"We have three recommendations to help improve these qualities in reasoning systems, all requiring high-quality data. A strong data partner like Toloka can help navigate the details of data production.\"}),/*#__PURE__*/e(\"h3\",{children:\"1) Diverse domain coverage\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Increasing versatility of autoverified data:\"}),\" Math and coding tasks permitting simple autoverification are the bread and butter of current reasoning systems [16], but there are ways to diversify.\"]}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Even math and coding have untapped niches. The U-MATH benchmark is derived from \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/datasets/university-level-math-reasoning-dataset\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"our larger training dataset\"})}),\", offering a good example of a \",/*#__PURE__*/e(\"strong\",{children:\"math subdomain that\u2019s underrepresented\"}),\" compared to the more readily-available textbook, high-school or Olympiad-style data.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Plenty of other fields allow for autoverification but suffer from a lack of appropriate data \u2014 examples include chemistry, finance, biology, and more. Closing the gap on these would require careful and efficient data curation by highly skilled experts from a diverse domain set \u2014 \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/experts\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Toloka\u2019s specialty\"})}),\".\"]})})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Expanding the boundaries of autoverification:\"}),\" Another direction for improvement would involve making better [17] or more general verifiers.\"]}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"One possible approach is to create \",/*#__PURE__*/e(\"strong\",{children:\"datasets for training more capable verifiers\"}),\" [18], such as scaling up meta-evaluation datasets like \",/*#__PURE__*/e(n,{href:\"https://huggingface.co/datasets/toloka/mu-math:\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Toloka\u2019s \u03BC-MATH\"})}),\" in size and complexity.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Alternatively, domains such as law or medicine don\u2019t typically have singular golden labels but lend themselves well to verification on \",/*#__PURE__*/e(\"strong\",{children:\"expert-formulated rubrics\"}),\", which Toloka already produces for a diverse set of complex domains.\"]})})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Explicit demonstration where autoverification fails:\"}),\" For more open-ended expert-reasoning domains, such as consulting, web research, or forecasting, proper autoverification is challenging to do at scale [19]. For these, a possible approach is to have expert-crafted demonstrations for the \",/*#__PURE__*/e(\"strong\",{children:\"entire reasoning process\"}),\" \u2014 planning, researching the sources, backtracking, etc. One way Toloka sets similar projects up is by having \",/*#__PURE__*/e(\"strong\",{children:\"experts operate an LLM-based system\"}),\", adding their inputs and steering it towards complete solutions.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Going Multi-*:\"}),\" multi-lingual, multi-cultural, multi-modal, multi-turn, etc.\"]}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Similar to niche subdomains, languages other than English are underrepresented in quality data. Our team has hands-on experience delivering \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/blog/multi-domain-multi-language-sft-dataset-pushes-llm-performance-to-the-next-level/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"production-grade multilingual datasets\"})}),\" and working with \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/blog/low-resource-languages/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"low-resource languages\"})}),\", thanks to our \",/*#__PURE__*/e(\"strong\",{children:\"expert network of speakers in 40+ languages\"}),\".\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Another option is producing \",/*#__PURE__*/e(\"strong\",{children:\"multimodal golden labels\"}),\" \u2014 e.g. for \",/*#__PURE__*/e(\"strong\",{children:\"reasoning over images\"}),\" \u2014 which \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/datasets/multimodal-dataset\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Toloka can do at scale\"})}),\".\"]})})]}),/*#__PURE__*/e(\"h3\",{children:\"2)\\xa0Reasoning refinement\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Process supervision:\"}),\" Training process reward models is a large part of the current reasoning systems research [20]. Despite the relevance, there\u2019s very little public data available for PRM training, and all the limitations of poor domain coverage apply here as well. This type of data production overlaps with Toloka\u2019s experience, such as \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/coding-data\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"trajectory annotation for coding agents\"})}),\" [21].\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Explicit demonstrations of excellent reasoning:\"}),\" Demonstration-style data could also be used to improve coherence, efficiency, and clarity of reasoning traces [22].\"]}),/*#__PURE__*/e(\"p\",{children:\"There are several ways to obtain this type of data:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Manually crafting examples of concise and effective reasoning from scratch\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Editing LLM-generated trajectories\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Issuing preferences amidst a number of alternative trajectory options\"})})]}),/*#__PURE__*/t(\"p\",{children:[\"All of these align with our experience of building \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/blog/llm-for-code-generation/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"scalable SFT demonstration pipelines\"})}),/*#__PURE__*/e(\"strong\",{children:\" \"}),\"for complex domains. The preference option could either be done over entire trajectories or provide denser signals in a step-wise manner, similar to Toloka\u2019s projects\",/*#__PURE__*/e(\"strong\",{children:\" \"}),\"in\",/*#__PURE__*/e(\"strong\",{children:\" multi-turn dialogue preferences.\"})]}),/*#__PURE__*/e(\"h3\",{children:\"3)\\xa0Appropriate evals\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Benchmarks:\"}),\" As we\u2019ve already emphasized, benchmarks need to be novel and provide clear signals \u2014 incorporating purposeful design, quality data and meaningful analyses. We have relevant experience in creating clean, informative, and practical evaluation datasets, such as our publicly available \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/math-benchmark\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"U-MATH & \u03BC-MATH\"})}),\", and in building \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/evaluation\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"custom evaluation pipelines\"})}),\" with insight into quality criteria and metrics tailored to specific use-cases.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Red-teaming:\"}),\" Traditional safety labeling will need to be adapted to reasoning models to take long reasoning traces into account. Besides that, as models become more prominent, they have broader applications and thus an expanded attack surface as well. Toloka\u2019s versatile team of \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/ai-safety\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"domain and security experts\"})}),\" tests a wide variety of applications, including \",/*#__PURE__*/e(\"strong\",{children:\"agentic systems\"}),\" and \",/*#__PURE__*/e(\"strong\",{children:\"computer operators\"}),\".\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"h2\",{children:\"The \u201CDeepSeek moment\u201D is still real\"}),/*#__PURE__*/e(\"p\",{children:\"DeepSeek\u2019s public release of R1 deserves recognition for the seminal AI moment that it is.\"}),/*#__PURE__*/t(\"p\",{children:[\"Although it is not a fully-open model in terms of reproduction transparency and, as we\u2019ve shown here, it is not quite at the frontier level, \",/*#__PURE__*/e(\"strong\",{children:\"this is the closest the AI community has ever been to an open frontier system\"}),\". Besides, this is an incredible opportunity for us to study and better understand what differentiates these models, what\u2019s potentially missing, and how to move forward.\"]}),/*#__PURE__*/e(\"p\",{children:\"Let\u2019s build the future together.\"})]});export const richText6=/*#__PURE__*/t(a.Fragment,{children:[/*#__PURE__*/e(\"h6\",{children:\"[1] Not fully open, since the data and code to reproduce the results are not published. Although the comprehensive technical reports accompanying DeepSeek\u2019s releases are extremely valuable and admirable.\"}),/*#__PURE__*/t(\"h6\",{children:[\"[2]\",/*#__PURE__*/e(\"strong\",{children:\" \"}),\"Note about the Arena benchmark: GPT-4o currently scores higher than both o1-preview and o1-mini, which is indicative of the benchmark\u2019s nature: it depends a lot on subjective human preferences aside from formal model capabilities.\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[3]\",/*#__PURE__*/e(\"strong\",{children:\" \"}),\"Note we're reporting the TextHard subset of our benchmark, omitting multi-modal problems and problems from the easier subjects \u2014 algebra and precalculus. We use neural verifiers to deduce correctness of solutions, which is different from the traditional LLM-as-a-judge setup in that the judge model is given a ground truth label to compare the evaluated solution's answer against. We employ an ensemble of three reasoning models acting as verifiers: o3-mini, R1, and gemini 2.0-flash-thinking.\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[4] Hisorically, we've been using single-pass accuracy under greedy decoding with an output token limit of 4096. For reasoners, we increase the limit to 32768 tokens. Following \",/*#__PURE__*/e(n,{href:\"https://github.com/deepseek-ai/DeepSeek-R1?tab=readme-ov-file#usage-recommendations\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"recommendations\"})}),\", we have also tested two separate Pass@1 setups for reasoning models by setting temperature to 0.3 and 0.6 (both with four rollouts and top_p=0.95). We observed little variation and only found negligible difference in terms of the final score compared to the greedy scheme, the latter one yielding slightly higher results. These comparisons have not been performed for o-series models since their API does not allow for greedy inference.\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[5] Due to the margings observed with MATH, U-MATH and AIME being quite small, here we're mostly speaking in terms of impressions rather than reliably identifiable score gaps. Most of the current LLM evaluations do not report any confidence intervals, due to proper uncertainty estimations being hard to track. \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2410.05229\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Some\"})}),\" \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2503.00137v1\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"works\"})}),\" do emerge, but no proven standard framework currently exists.\\xa0\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[6] For more complex math subjects, \",/*#__PURE__*/e(n,{href:\"https://framerusercontent.com/images/Ma65VDHh57Gfwp9fFR5hBtP6Oxs.png\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"this is a truly demanding task\"})}),\".\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[7]\",/*#__PURE__*/e(\"strong\",{children:\" \"}),\"Note that arriving at the correct final judgment can be viewed as a classification problem: predicting a True / False label that's compared to the golden judge verdict. We thus use F1-score \u2014 a standard choice of a classification quality measure \u2014 as our primary metric. Note that unlike U-MATH, neural verifiers are not required here, since ground-truth comparisons are far less complex.\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[8] GPT4 is \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2312.09390#subsection.A.2\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"known to have been trained\"})}),\" on chess data. Whether chess data is used in training the DeepSeek models is not yet known. Howether, it is worth noting that the data in question is based on real game logs in PGN notation, while LLM Chess leaderboard involves issuing moves in an interactive manner based on graphical representations of board states while playing against a randomly moving bot. It is also worth noting that GPT4 does not get even a single win with this setting, same as DeepSeek-V3, and across the leaderboard only the reasoning systems are able to arrive at non-trivial numbers of wins.\\xa0\"]}),/*#__PURE__*/e(\"h6\",{children:\"[9] MathArena authors report some of the problems having identical or presumably similar ones available on the popular web resources such as Quora and Math StackExchange, thus making exposure possible. However, for similar problems it is unclear whether resemblance is high enough to meaningfully affect models' capabilities in solving the original ones. Apart from that, most of the sources where the exposed problems are found do not contain correct solutions to them.\"}),/*#__PURE__*/t(\"h6\",{children:[\"[10] In principle, test-time compute should help with safety \u2014 there are even some reports on this already, e.g. by OpenAI with their \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2501.18841\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"study of o-series models\"})}),\" \u2014 but that doesn\u2019t seem to happen for Deepseek at all.\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[11] It's fascinating that \",/*#__PURE__*/e(\"strong\",{children:\"Gemini 1.5 Pro aligns more closely with reasoning systems\"}),\" than with other standard inference models \u2014 a remarkable achievement at the time. Did the team already have traces from an experimental in-development reasoning system to distill from? We\u2019re not sure, especially considering the uniquely word-efficient, succinct responses \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/html/2412.03205v3/x3.png\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Gemini demonstrated on U-MATH\"})}),\". In light of this, our bet is on high-quality guidance, similar to what we\u2019ve discussed for o1.\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[12] We do that in our \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2412.03205\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"research paper\"})}),\".\"]}),/*#__PURE__*/e(\"h6\",{children:\"[13] And also forming distinctive clusters of model families, with Qwens beings more lenient and proprietary models being more conservative. This reminds that when referring to a model, one is referring in large part to its training data.\"}),/*#__PURE__*/t(\"h6\",{children:[\"[14]\",/*#__PURE__*/e(\"strong\",{children:\" \"}),\"Mostly pushing to the right side of the chart, consistent with our observations that increased mathematical problem-solving and verbosity \u2014 hallmarks of current reasoner systems \u2014 correlate with an increase in true positive rate.\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[15]\",/*#__PURE__*/e(\"strong\",{children:\" \"}),\"Notably, \",/*#__PURE__*/e(\"strong\",{children:\"R1 is the only reasoning system that is closer to conservative models\"}),\" in terms of its scores. Upon inspection, we found that its reasoning traces are indeed often driving it towards conservative judgments, the model displaying \u201Chyper-fixation\u201D over minute details of the golden labels. This is the first case we encountered where an increase in coherence would probably aid more with true positives rather than true negatives. But the sentiment remains the same: coherence and reliability are required to appropriately and successfully apply the problem-solving skills to the task at hand.\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[16]\",/*#__PURE__*/e(\"strong\",{children:\" \"}),\"That is because objectively verifiable tasks are perfect for applying the classical reinforcement learning framework to LLM training i\",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2410.01679#section.2\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"n a robust manner.\"})})]}),/*#__PURE__*/e(\"h6\",{children:\"[17] This quickly becomes a necessity even with formal domains such as math when considering the more challenging tasks, as illustrated by the \u03BC-MATH benchmark. Transitioning into domains such as compliance further exacerbates the problem.\"}),/*#__PURE__*/t(\"h6\",{children:[\"[18] The current consensus is to avoid neural verifiers due to the potential for reward hacking, but there are already proof-of concepts that avoid this risk, such as \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2412.16339\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"OpenAI\u2019s safety training for o-series of models\"})}),\".\"]}),/*#__PURE__*/e(\"h6\",{children:\"[19] Current models are also inept at generating quality post-hoc reasoning traces for such tasks.\"}),/*#__PURE__*/t(\"h6\",{children:[\"[20] Pioneering work in the LLM space here is \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2305.20050\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"OpenAI\u2019s PRM800k\"})}),\", a dataset of automatic math solutions with each step labeled by a human annotator. Qwen \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2501.07301\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"report using a similar approach\"})}),\" for their QwQ reasoner model, \",/*#__PURE__*/e(\"strong\",{children:\"specifically advocating for human labeling over the supervision-free, autoverification-based approaches\"}),\". While Gemini \",/*#__PURE__*/e(n,{href:\"https://arxiv.org/pdf/2406.06592v1\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"report\"})}),\" relying on exactly such data generation methods, they acknowledge running into the usual limitations of narrowing to autoverifiable tasks only.\"]}),/*#__PURE__*/t(\"h6\",{children:[\"[21] When viewing reasoning system training from the standpoint of classical reinforcement learning, such process supervising data could be considered a form of \",/*#__PURE__*/e(n,{href:\"https://www.sci.brooklyn.cuny.edu/~sklar/teaching/boston-college/s01/mc375/ml94.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"reward shaping\"})}),\".\"]}),/*#__PURE__*/e(\"h6\",{children:\"[22] An example is given with the DeepSeek-R1 technical report, where they refer to this as \u201Chuman priors\u201D.\"})]});export const richText7=/*#__PURE__*/t(a.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"GenAI progress now depends on sophisticated data for training and evaluating models, and high-value data is in short supply. To save on data production costs, general human knowledge can be replicated by LLMs to produce synthetic data for these purposes. But synthetic data inherits biases and shows limitations when it comes to specialized knowledge.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"There isn\u2019t a full substitute for human-generated data when fairness, accountability, and safety are a priority. GenAI development still demands unique data contributed by human experts. As AI models become more specialized, the demand for human-generated data is shifting to niche domains where general knowledge is insufficient, paving the way for a new role: the AI Tutor.\"}),/*#__PURE__*/e(\"p\",{children:\"At Toloka, we recruit AI Tutors to craft custom datasets for AI post-training. These professionals are experienced writers, editors, and experts in highly specialized fields, from coding to law to linguistics. The strong skillsets and deep expertise of our AI Tutors are instrumental in refining outputs, mitigating bias, and ensuring models align with ethical and regulatory standards.\"}),/*#__PURE__*/e(\"h2\",{children:\"Introducing Mindrift\"}),/*#__PURE__*/t(\"p\",{children:[\"To meet the industry's growing demand for specialized talent and domain-specific knowledge, last year we launched the \",/*#__PURE__*/e(n,{href:\"https://mindrift.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:/*#__PURE__*/e(\"strong\",{children:\"Mindrift\"})})}),\" platform, a hub for data production where part-time AI Tutors can contribute to datasets for AI projects focused on their areas of expertise. Leveraging over a decade of experience in building world-class crowdsourcing platforms for AI training, we have refined our data production technology to deliver a seamless and enriching experience for AI Tutors.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Mindrift is a scalable, end-to-end solution for creating custom datasets from scratch.\\xa0 The platform integrates seamlessly with Toloka to provide clients with curated data crafted by highly educated professionals. Expert knowledge is integrated into data production pipelines that ensure high-quality outputs, supporting critical processes such as \",/*#__PURE__*/e(\"strong\",{children:\"supervised fine-tuning\"}),\", \",/*#__PURE__*/e(\"strong\",{children:\"reinforcement learning with human feedback\"}),\", \",/*#__PURE__*/e(\"strong\",{children:\"model evaluation, and red-teaming\"}),\".\"]}),/*#__PURE__*/e(\"h2\",{children:\"Opportunities for AI Tutors\"}),/*#__PURE__*/e(\"p\",{children:\"Mindrift\u2019s global reach fosters diversity in datasets while empowering individuals worldwide. We source data from a diverse range of contributors across geographies, cultures, and languages.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"For many, this represents an opportunity to apply their expertise in a dynamic and cutting-edge field while earning extra income. Whether navigating a career break or adding a side project, AI Tutors join Mindrift for the flexible and challenging opportunity to do something new.\"}),/*#__PURE__*/e(\"blockquote\",{children:/*#__PURE__*/t(\"p\",{children:[\"Mindrift has provided me with the most interesting, challenging and flexible work I've ever done. Looking back on my year-long journey, I see only good things on the horizon. It's exciting to know I'm making a difference in the future of AI. \u2013 \",/*#__PURE__*/e(\"strong\",{children:\"Lisa, AI Tutor\\xa0\"})]})}),/*#__PURE__*/e(\"blockquote\",{children:/*#__PURE__*/t(\"p\",{children:[\"Freelancing has always been a rollercoaster, but the ride is much smoother now that I can precisely pick my hours and don't rely on project managers or clients. I'm grateful for the flexible earning ability I have now, and I hope that my story encourages others interested in this type of work to reach out and see how they can get started \u2013 \",/*#__PURE__*/e(\"strong\",{children:\"Christopher, Editor\"})]})}),/*#__PURE__*/e(\"h2\",{children:\"Ensuring quality\"}),/*#__PURE__*/e(\"p\",{children:\"As we leverage human intelligence to shape AI, we hold AI Tutors to high standards for data quality.\\xa0\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Our robust security system is designed to detect and prevent fraud, guarding the integrity of the datasets we produce.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"We work to ensure quality at every stage of data production, with ongoing quality checks platform-wide.\\xa0\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"We develop tailored quality workflows for each project based on client requirements.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"AI Tutors are individually vetted and their performance is tracked.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"AI Tutors are supported with smooth onboarding, targeted training, and ongoing collaboration to continually boost quality.\\xa0\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"The journey of an AI Tutor on the Mindrift platform\"}),/*#__PURE__*/e(\"p\",{children:\"Before joining Mindrift, applicants complete testing to demonstrate in-depth knowledge of their field. This process includes both writing tests and domain-specific tests.\"}),/*#__PURE__*/e(\"p\",{children:\"Each project on the Mindrift platform has its own onboarding process, including a training course and exams, to help AI Tutors master the project requirements before they start taking tasks. We monitor the pass rates and regularly review the onboarding materials to calibrate the tests and make onboarding as smooth as possible.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Every expert is additionally reviewed during active participation on a project. A portion of their work is vetted and measured with quality metrics. If quality drops, we suspend the expert\u2019s access to the project while we review their tasks and offer mentoring to resolve issues.\\xa0\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Quality Assurance specialists are assigned to the project teams to ensure that completed tasks meet project guidelines. These experts have the highest qualifications in their field and in-depth knowledge of project requirements, so their job is to review tasks with a sharp eye and provide constructive feedback to AI Tutors.\"}),/*#__PURE__*/e(\"h2\",{children:\"Prioritizing the AI Tutor experience\"}),/*#__PURE__*/e(\"p\",{children:\"Our top priority is to attract the best talent to Mindrift and keep AI Tutors coming back for more projects that match their skills and interests. We focus on giving our experts the best possible experience and producing the best possible data.\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Competitive pay and convenient payment systems, with flexibility in time commitments, and the freedom to opt out of tasks.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"A technological platform with co-pilots to automate routine tasks for AI Tutors, increasing productivity by 45% and eliminating repetitive workloads, preventing burnout among our experts.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"A supportive environment focused on collaboration and community. AI Tutors use our Discord community to get quick support, communicate with peers, and stay updated on the latest platform developments. Experts can share knowledge and insights to generate better content and accelerate professional growth.\"})})]}),/*#__PURE__*/e(\"blockquote\",{children:/*#__PURE__*/t(\"p\",{children:[\"Mindrift is about collaboration and growth. The Discord community allows for focused chats, organized teams, and a platform for asking questions. It's a supportive environment where I can engage with other AI Tutors and learn from my mistakes. \u2013 \",/*#__PURE__*/e(\"strong\",{children:\"Lance, AI Tutor\"})]})}),/*#__PURE__*/t(\"p\",{children:[\"With the launch of \",/*#__PURE__*/e(n,{href:\"https://mindrift.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:/*#__PURE__*/e(\"strong\",{children:\"Mindrift\"})})}),\", we are not just building datasets for GenAI models\u2014we are creating opportunities, advancing technology, and empowering individuals to shape the future of AI.\\xa0\"]}),/*#__PURE__*/t(\"p\",{children:[\"Combining synthetic and real-world data leads to more reliable AI systems, and this is where Toloka excels. Do you need expert knowledge to improve your LLM? \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/experts\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Reach out\"})}),\" to us. We have experts in more than 20 domains, including STEM, coding, law, finance, automotive, ESG, medicine, manufacturing, civil engineering, linguistics, and more.\"]})]});export const richText8=/*#__PURE__*/t(a.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"In recent weeks, DeepSeek has shaken the AI world, with discussions spreading across mainstream media, researchers, AI developers, tech enthusiasts, and industry leaders. Once a relatively unknown player in the LLM space, their latest model, \",/*#__PURE__*/e(n,{href:\"https://api-docs.deepseek.com/news/news250120\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"DeepSeek R1\"})}),\", has matched the best existing LLM models on several popular leaderboards. Additionally, DeepSeek R1 is published under the MIT license, and a technical report accompanied its release. This achievement is even more remarkable because they claim the model was trained on a budget of just $5.6 million, a fraction of what competitors have spent on similar models.\"]}),/*#__PURE__*/e(\"p\",{children:\"So, what\u2019s the secret behind DeepSeek\u2019s success? The technical report leaves out key details, particularly regarding data collection and training methodologies.\"}),/*#__PURE__*/t(\"p\",{children:[\"Several open-source initiatives, such as the \",/*#__PURE__*/e(n,{href:\"https://huggingface.co/blog/open-r1\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Open-R1\"})}),\" project on Hugging Face, are now working to reproduce DeepSeek R1. In this article, Toloka\u2019s researchers analyze the key factors that set DeepSeek R1 apart and explore the data requirements for building your own R1 model, or an even better version.\"]}),/*#__PURE__*/e(\"h2\",{children:\"What did DeepSeek do differently?\"}),/*#__PURE__*/e(\"p\",{children:\"DeepSeek\u2019s success with R1 comes from rethinking the standard training process. Traditionally, large models undergo supervised fine-tuning (SFT) first, followed by reinforcement learning (RL) for alignment and tuning on complex tasks. DeepSeek revised this approach.\"}),/*#__PURE__*/e(\"p\",{children:\"Instead of fine-tuning first, they applied RL with math and coding tasks early in training to enhance reasoning abilities. This allowed the model to generate answers independently with minimal supervision, only validating the final answer, and maximizing the benefits of pre-training for reasoning. The model\u2019s skills were then refined and expanded beyond the math and coding domains through fine-tuning for non-reasoning tasks.\"}),/*#__PURE__*/t(\"p\",{children:[\"While this provides a high-level understanding of DeepSeek\u2019s approach, it\u2019s important to examine the data used at each stage of training. The following diagram breaks down the key training steps in more detail.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"1097\",src:\"https://framerusercontent.com/images/LgJgIDWxRDTmxd1ZASUemthgfys.png\",srcSet:\"https://framerusercontent.com/images/LgJgIDWxRDTmxd1ZASUemthgfys.png?scale-down-to=512 512w,https://framerusercontent.com/images/LgJgIDWxRDTmxd1ZASUemthgfys.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/LgJgIDWxRDTmxd1ZASUemthgfys.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/LgJgIDWxRDTmxd1ZASUemthgfys.png 2560w\",style:{aspectRatio:\"2560 / 2194\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"DeepSeek-R1 training pipeline\"})}),/*#__PURE__*/e(\"h3\",{children:\"Stage 1: Cold Start (SFT)\"}),/*#__PURE__*/e(\"p\",{children:\"Using a small LLM-generated and human-curated dataset of demonstrations, the model was first trained on high-quality reasoning data (math and code). These examples focused on improving the consistency and readability of reasoning trajectories rather than enhancing reasoning ability itself. This phase helped accelerate convergence in the following reinforcement learning (RL) stage.\\xa0\"}),/*#__PURE__*/e(\"h3\",{children:\"Stage 2: Reasoning-Oriented RL\"}),/*#__PURE__*/e(\"p\",{children:\"This stage provided the biggest performance boost. The model was trained on tasks with auto-verifiable answers (math, code, logic) using predefined rule-based checks as the primary reward signal. No human demonstrations were included, only deterministic correctness checks (e.g., math answer exact-match) and rule-based evaluations for reasoning format and language consistency. While format checks slightly constrained performance, it ensured more human-friendly reasoning outputs.\"}),/*#__PURE__*/e(\"h3\",{children:\"Stage 3: Synthetic SFT\"}),/*#__PURE__*/e(\"p\",{children:\"DeepSeek used synthetic data to fine-tune the model. Specifically, 600,000 reasoning data samples were generated through rejection sampling and refinement from the RL-trained model described above, and 200,000 non-reasoning data samples were derived from DeepSeek-V3, covering writing, QA, and translation tasks. In total, 800,000 samples were used to fine-tune the base model.\"}),/*#__PURE__*/e(\"h3\",{children:\"Stage 4: Mixed RL (Reasoning + RLHF)\"}),/*#__PURE__*/e(\"p\",{children:\"At this final stage, auto-verifiable rule-based rewards continued to refine reasoning tasks, while preference-based RLHF (similar to DeepSeek-V3) was applied to general tasks. The final results were optimized for helpfulness, while both reasoning chains and results were tuned for safety.\"}),/*#__PURE__*/e(\"h2\",{children:\"How to replicate DeepSeek\u2019s process by focusing on the right data\"}),/*#__PURE__*/e(\"p\",{children:\"The most significant performance boost in DeepSeek R1 came from reasoning-oriented RL. To replicate or exceed their success, prioritize high-quality data for this stage. They used auto-verifiable tasks such as math and coding, where answers are clearly defined and can be automatically checked (e.g., through unit tests or predetermined answers). While DeepSeek concentrated on math and coding, this approach can be extended to other domains, such as physics or chemistry, where automatic verification is possible.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"However, other types of data are also essential. Invest in high-quality chain-of-thought demonstrations designed for cold-start reasoning training for further improvement. Rather than relying on generic chain-of-thought data, target specific domains or languages to achieve the best performance boost. Additionally, include classic SFT data for non-auto-verifiable tasks and human preferences for final model alignment.\"}),/*#__PURE__*/e(\"h2\",{children:\"Beyond DeepSeek performance\"}),/*#__PURE__*/t(\"p\",{children:[\"At first glance, based on common benchmarks, DeepSeek R1 appears to perform similarly to OpenAI\u2019s reasoning model o1. It slightly outperforms o1 in reasoning tasks (e.g., Math 500, SWE Verified) and falls just behind in general knowledge benchmarks (MMLU, Simple QA). However, the performance gap becomes more noticeable in niche and out-of-domain areas. For example, on the \",/*#__PURE__*/e(n,{href:\"https://maxim-saplin.github.io/llm_chess/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"LLM Chess leaderboard\"})}),\", o1 achieves a 46.67% win rate, while R1 reaches only 22.58%.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Toloka\u2019s researchers have conducted additional tests on \",/*#__PURE__*/e(n,{href:\"https://toloka.ai/math-benchmark\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"U-MATH\"})}),\", a dataset of complex university-level mathematics, where R1 performed significantly worse than o1. (We will publish and link the detailed analysis soon.)\"]}),/*#__PURE__*/e(\"p\",{children:\"Why does o1 perform better in these specialized areas? Is DeepSeek R1 truly strong in mathematics? While R1 outperforms o1 on MATH-500, it struggles with more advanced university-level problems. A likely explanation is that MATH-500 includes data within R1\u2019s training distribution, whereas U-MATH contains out-of-domain challenges. DeepSeek R1 was trained on widely available datasets that do not include advanced, proprietary mathematical problems.\"}),/*#__PURE__*/e(\"p\",{children:\"To surpass DeepSeek R1, we recommend incorporating complex, domain-specific data. Training on widely available datasets limits a model\u2019s ability to handle novel, specialized tasks. By integrating high-quality data from niche fields, you can develop a model that excels where R1 currently falls short.\"}),/*#__PURE__*/e(\"h2\",{children:\"Are you ready to take your model to the next level?\"}),/*#__PURE__*/e(\"p\",{children:\"Partner with Toloka to take your model performance to the next level. We offer top-tier Auto-Verifiable Tasks, similar to those used in DeepSeek RL training, designed to enhance objective reasoning through automated feedback. Our experts create complex prompts, test cases, answers, and rubrics to ensure precision and reliability.\"}),/*#__PURE__*/e(\"p\",{children:\"Additionally, we provide specialized domain-specific SFT data, chain-of-thought SFT data, and human feedback for final model alignment.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(n,{href:\"https://toloka.ai/talk-to-us\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(i.a,{children:\"Connect with our team\"})}),\" to explore your data needs. \"]})]});\nexport const __FramerMetadata__ = {\"exports\":{\"richText5\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText3\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText1\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText8\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText7\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText2\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText4\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText6\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"__FramerMetadata__\":{\"type\":\"variable\"}}}"],
  "mappings": "6JAAsJ,IAAMA,EAAsBC,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,qEAAqE,OAAO,6VAA6V,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8aAA8a,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,4BAA4B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,qUAAgU,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,iPAA4O,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,0DAAuEE,EAAE,SAAS,CAAC,SAAS,uEAAuE,CAAC,EAAE,sXAAsX,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ymBAAymB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iMAA4L,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sFAAsF,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,kMAAkM,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,0CAA0C,CAAC,EAAE,yPAAyP,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uCAAuC,CAAC,EAAE,4VAAuV,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,+VAAqV,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4CAA4C,CAAC,EAAE,iNAA8NA,EAAEC,EAAE,CAAC,KAAK,0GAA0G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,iWAAiW,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,0GAAuHE,EAAEC,EAAE,CAAC,KAAK,iCAAiC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,EAAE,oMAA+L,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,mXAAmX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0jBAAqjB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,OAAoBE,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,EAAE,mFAAmF,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,kDAAkD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qMAAqM,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,EAAE,iJAAiJ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,wIAAwI,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,EAAE,yJAAyJ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iZAAiZ,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yFAAyF,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oeAAoe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uFAAuF,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,+JAA+J,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,2JAA2J,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qIAAqI,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6CAA6C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4WAA4W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mNAAmN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6QAA6Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kTAAkT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0DAA0D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uSAAuS,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,kZAA6Y,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,EAAE,uTAAuT,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,8CAA8C,CAAC,EAAE,+WAAqW,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,+JAA+J,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2CAA2C,CAAC,EAAE,uQAAuQ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oDAAoD,CAAC,EAAE,4VAAyWA,EAAEC,EAAE,CAAC,KAAK,gFAAgF,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,gBAAgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+fAA2e,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,MAAmBE,EAAE,SAAS,CAAC,SAAS,QAAQ,CAAC,EAAE,oRAAoR,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2DAA2D,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,8EAA2FE,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeC,EAAuBL,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,+GAA4HE,EAAEC,EAAE,CAAC,KAAK,8EAA8E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,EAAE,2aAAsa,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,idAAid,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6RAA6R,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAyB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kXAAkX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oHAAoH,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,+BAA+B,CAAC,EAAE,2XAA2X,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oCAAoC,CAAC,EAAE,wTAAwT,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2cAAub,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qYAAqY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8CAA8C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yQAAyQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,maAA8Z,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2CAA2C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4OAAuO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2WAAsW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,geAA2d,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qMAAgM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+CAA+C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yVAAyV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4eAA8c,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gRAAgR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ycAA0b,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+CAA+C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4XAAuX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2QAAsQ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wPAAwP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAA0W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wXAAwX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uDAAuD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iUAAuT,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,6FAA6F,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,yEAAyE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,gIAA2H,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yLAAyL,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sDAAsD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gPAAgP,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,kFAAkF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,yHAAyH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,wGAAwG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kcAAkc,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oDAAoD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6QAAwQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qUAAgU,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0DAA0D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uVAAuV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6RAA6R,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4hBAA4hB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6MAA6M,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sVAAsV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8SAAoS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8hBAA8hB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0XAA0X,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sYAAsY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6RAA6R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qNAAqN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,siBAAsiB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wPAAmP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+DAA0D,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oDAAoD,CAAC,EAAE,iMAAiM,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qCAAqC,CAAC,EAAE,0LAA0L,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mKAA8J,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAyB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oPAAoP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mWAAmW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mXAA8W,CAAC,CAAC,CAAC,CAAC,EAAeI,EAAuBN,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,MAAmBE,EAAE,SAAS,CAAC,SAAS,+BAA+B,CAAC,EAAE,0IAAuJA,EAAE,SAAS,CAAC,SAAS,4CAA4C,CAAC,EAAE,8PAA+O,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4CAA4C,CAAC,EAAE,oCAAiDA,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,EAAE,YAAyBA,EAAE,SAAS,CAAC,SAAS,4CAA4C,CAAC,EAAE,eAA4BA,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,+GAA+G,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ufAAwe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qeAAge,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8pBAA8pB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qOAAkPE,EAAEC,EAAE,CAAC,KAAK,uBAAuB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,kKAAkK,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ozBAA+yB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oDAAoD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAoX,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,0GAA0G,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kNAAkN,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,mEAA8D,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,EAAE,6NAAwN,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kCAAkC,CAAC,EAAE,gFAA2E,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,gKAA2J,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uWAA6V,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8CAA8C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2OAA2O,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,4DAA4D,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mhBAAmhB,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8OAA8O,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,yKAAsLE,EAAEC,EAAE,CAAC,KAAK,+BAA+B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4BAA4B,CAAC,CAAC,CAAC,EAAE,4NAAuN,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,+BAA+B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAE,+BAA+B,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeG,EAAuBP,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,iIAA4H,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iQAAiQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kLAAkL,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+CAA+C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6PAA6P,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,8KAAoK,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,sHAAsH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,+NAA+N,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,mKAAmK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,+BAA+B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,+CAA+C,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,0DAA0D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uLAAuL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iUAAiU,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,6HAA6H,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,gPAAgP,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,2HAA2H,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,gMAAgM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yHAAyH,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wBAAqCE,EAAEC,EAAE,CAAC,KAAK,uBAAuB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,YAAY,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oSAAoS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iFAAiF,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,wMAAwM,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,6IAA6I,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2DAA2D,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,0JAA0J,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,8EAA8E,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,0KAA0K,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+EAA4FE,EAAEC,EAAE,CAAC,KAAK,oCAAoC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,sDAAsD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qTAAqT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sMAAsM,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,iEAAiE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,2IAA2I,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,yGAAyG,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,sKAAsK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4FAAyGE,EAAEC,EAAE,CAAC,KAAK,6BAA6B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,6CAA6C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kOAAkO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2CAA2C,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,2IAA2I,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,oIAAoI,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,kIAAkI,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,8KAA8K,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qBAAkCE,EAAEC,EAAE,CAAC,KAAK,6BAA6B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gOAAgO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gFAAgF,CAAC,CAAC,CAAC,CAAC,EAAeM,EAAuBR,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,8WAAoW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0SAA0S,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wVAAwV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4SAAuS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uCAAuC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yKAAyK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mVAAmV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sWAAsW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,0IAAuJE,EAAEC,EAAE,CAAC,KAAK,kCAAkC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAE,6DAA6D,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,yEAAoE,CAAC,CAAC,CAAC,CAAC,EAAeO,EAAuBT,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,+EAA+E,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uDAAuD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,mDAAgEE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,6IAA6I,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,8PAA4PE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,+dAA+d,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0NAAqN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,mJAA8I,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,oDAA+C,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,gEAAgE,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAS,CAAcE,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,iCAAiC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uDAAuD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qMAAqM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4OAAuO,CAAC,EAAeA,EAAE,SAAS,CAAC,UAAU,uBAAuB,SAAsBA,EAAE,QAAQ,CAAC,SAAsBF,EAAE,QAAQ,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,+BAA+B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gEAAmEE,EAAEC,EAAE,CAAC,KAAK,wBAAwB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,qCAAqC,CAAC,CAAC,EAAeF,EAAE,SAAS,CAAC,UAAU,uBAAuB,SAAsBA,EAAE,QAAQ,CAAC,SAAsBF,EAAE,QAAQ,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,oCAAoC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,0LAAwLE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,IAAiBF,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAoX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,weAA0e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,8CAAyC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qPAAqP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qCAAkDE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,wUAA2UF,EAAE,SAAS,CAAC,SAAS,qEAAqE,CAAC,EAAE,YAAY,CAAC,CAAC,EAAeA,EAAE,SAAS,CAAC,UAAU,uBAAuB,SAAsBA,EAAE,QAAQ,CAAC,SAAsBF,EAAE,QAAQ,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+JAA4KE,EAAEC,EAAE,CAAC,KAAK,qEAAqE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAkB,CAAC,CAAC,CAAC,EAAE,yMAAsNF,EAAE,SAAS,CAAC,SAAS,iDAAiD,CAAC,EAAE,yCAAyC,CAAC,CAAC,EAAeA,EAAE,SAAS,CAAC,UAAU,uBAAuB,SAAsBA,EAAE,QAAQ,CAAC,SAAsBF,EAAE,QAAQ,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,+BAA+B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8NAA8N,CAAC,EAAeA,EAAE,SAAS,CAAC,UAAU,uBAAuB,SAAsBA,EAAE,QAAQ,CAAC,SAAsBF,EAAE,QAAQ,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAa,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,MAAM,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAQ,CAAC,CAAC,CAAC,EAAE,8TAAsUF,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,EAAE,4IAAuI,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,wDAAwD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,KAAK,CAAC,CAAC,CAAC,EAAE,qGAAqG,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,4CAA4C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAE,+NAA+N,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wDAAwD,CAAC,CAAC,CAAC,EAAE,6gBAA6gB,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,iCAA8CE,EAAEC,EAAE,CAAC,KAAK,+CAA+C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAE,oGAAoG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,yEAAyE,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wDAAqEE,EAAE,SAAS,CAAC,SAAS,6DAA6D,CAAC,EAAE,+BAAuCA,EAAE,SAAS,CAAC,SAAS,uDAAuD,CAAC,EAAE,kGAAkG,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2GAA2G,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iPAA8PE,EAAEC,EAAE,CAAC,KAAK,wBAAwB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,8MAAyM,CAAC,CAAC,EAAeF,EAAE,SAAS,CAAC,UAAU,uBAAuB,SAAsBA,EAAE,QAAQ,CAAC,SAAsBF,EAAE,QAAQ,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oIAAoI,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,4GAA4G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,gHAA6HF,EAAEC,EAAE,CAAC,KAAK,gFAAgF,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAE,IAAiBF,EAAEC,EAAE,CAAC,KAAK,oIAAoI,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAE,wCAAmC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAsBA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,YAAyBE,EAAEC,EAAE,CAAC,KAAK,uDAAuD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6CAA6C,CAAC,CAAC,CAAC,EAAE,yGAAyG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,iMAA8ME,EAAE,SAAS,CAAC,SAAS,8DAA8D,CAAC,EAAE,4QAA4Q,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,6GAA0HE,EAAE,SAAS,CAAC,SAAS,4DAA4D,CAAC,EAAE,kBAAkB,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6CAA6C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kQAA6P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wDAAgEE,EAAEC,EAAE,CAAC,KAAK,sHAAsH,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,4DAAuD,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,qEAAqE,OAAO,oKAAoK,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mDAAmD,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,+BAA+B,CAAC,EAAE,qZAAgZ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sCAAsC,CAAC,EAAE,6IAA6I,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,yEAAyE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mOAA8N,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,8LAAyL,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,+KAA0K,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,0JAAuKE,EAAE,SAAS,CAAC,SAAS,yCAAyC,CAAC,EAAE,sBAAmCA,EAAE,SAAS,CAAC,SAAS,gCAAgC,CAAC,EAAE,uCAAuC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8TAA8T,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,yKAAyK,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qHAAkIE,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,EAAE,qKAAkLA,EAAE,SAAS,CAAC,SAAS,oCAAoC,CAAC,EAAE,4DAA4D,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wGAAwG,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mDAAmD,CAAC,EAAE,gQAAgQ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uDAAuD,CAAC,EAAE,+PAA0P,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,EAAE,qOAAqO,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4EAAuE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mMAAyL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0MAA0M,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,8CAA8C,CAAC,EAAE,wJAAwJ,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,mFAAgGE,EAAEC,EAAE,CAAC,KAAK,qEAAqE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAE,kCAA+CF,EAAE,SAAS,CAAC,SAAS,6CAAwC,CAAC,EAAE,uFAAuF,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,qSAAwSE,EAAEC,EAAE,CAAC,KAAK,4BAA4B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yBAAoB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,+CAA+C,CAAC,EAAE,gGAAgG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,sCAAmDE,EAAE,SAAS,CAAC,SAAS,8CAA8C,CAAC,EAAE,2DAAwEA,EAAEC,EAAE,CAAC,KAAK,kDAAkD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,2BAAiB,CAAC,CAAC,CAAC,EAAE,0BAA0B,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,+IAAuJE,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,EAAE,uEAAuE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sDAAsD,CAAC,EAAE,gPAA6PA,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,EAAE,sHAA8HA,EAAE,SAAS,CAAC,SAAS,qCAAqC,CAAC,EAAE,mEAAmE,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,+DAA+D,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,+IAA4JE,EAAEC,EAAE,CAAC,KAAK,2GAA2G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,EAAE,qBAAkCF,EAAEC,EAAE,CAAC,KAAK,iDAAiD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAE,mBAAgCF,EAAE,SAAS,CAAC,SAAS,6CAA6C,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,+BAA4CE,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,EAAE,oBAA4BA,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,iBAAyBA,EAAEC,EAAE,CAAC,KAAK,gDAAgD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,4UAA+UA,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yCAAyC,CAAC,CAAC,CAAC,EAAE,QAAQ,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iDAAiD,CAAC,EAAE,sHAAsH,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qDAAqD,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4EAA4E,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oCAAoC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uEAAuE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,sDAAmEE,EAAEC,EAAE,CAAC,KAAK,kDAAkD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sCAAsC,CAAC,CAAC,CAAC,EAAeF,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,8KAAsLA,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,KAAkBA,EAAE,SAAS,CAAC,SAAS,mCAAmC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAyB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,wSAA2SA,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sBAAiB,CAAC,CAAC,CAAC,EAAE,qBAAkCF,EAAEC,EAAE,CAAC,KAAK,+BAA+B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAE,iFAAiF,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,mRAA2RA,EAAEC,EAAE,CAAC,KAAK,8BAA8B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAE,oDAAiEF,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,QAAqBA,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+CAAqC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iGAA4F,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qJAA6JE,EAAE,SAAS,CAAC,SAAS,+EAA+E,CAAC,EAAE,gLAA2K,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uCAAkC,CAAC,CAAC,CAAC,CAAC,EAAeQ,EAAuBV,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAS,kNAA6M,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,MAAmBE,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,6OAAwO,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,MAAmBE,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,ofAA+e,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,oLAAiME,EAAEC,EAAE,CAAC,KAAK,sFAAsF,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,wbAAwb,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAC,0TAAuUE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,MAAM,CAAC,CAAC,CAAC,EAAE,IAAiBF,EAAEC,EAAE,CAAC,KAAK,qCAAqC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,oEAAoE,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAC,uCAAoDE,EAAEC,EAAE,CAAC,KAAK,uEAAuE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAC,MAAmBE,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,gZAAsY,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,eAA4BE,EAAEC,EAAE,CAAC,KAAK,kDAAkD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4BAA4B,CAAC,CAAC,CAAC,EAAE,mkBAAmkB,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,wdAAwd,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,8IAAsJE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAE,mEAAyD,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAC,8BAA2CE,EAAE,SAAS,CAAC,SAAS,2DAA2D,CAAC,EAAE,8RAAiSA,EAAEC,EAAE,CAAC,KAAK,6CAA6C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,+BAA+B,CAAC,CAAC,CAAC,EAAE,uGAAkG,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAC,0BAAuCE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,gPAAgP,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,OAAoBE,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,iPAAuO,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,OAAoBE,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,YAAyBA,EAAE,SAAS,CAAC,SAAS,uEAAuE,CAAC,EAAE,ohBAA0gB,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,OAAoBE,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,yIAAsJA,EAAEC,EAAE,CAAC,KAAK,6CAA6C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,sPAAiP,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,0KAAuLE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sDAAiD,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,oGAAoG,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,iDAA8DE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAkB,CAAC,CAAC,CAAC,EAAE,6FAA0GF,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iCAAiC,CAAC,CAAC,CAAC,EAAE,kCAA+CF,EAAE,SAAS,CAAC,SAAS,yGAAyG,CAAC,EAAE,kBAA+BA,EAAEC,EAAE,CAAC,KAAK,qCAAqC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,kJAAkJ,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAC,oKAAiLE,EAAEC,EAAE,CAAC,KAAK,sFAAsF,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,uHAA6G,CAAC,CAAC,CAAC,CAAC,EAAeS,EAAuBX,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,qWAAqW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8XAAyX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oYAAoY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,yHAAsIE,EAAEC,EAAE,CAAC,KAAK,uBAAuB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,qWAAqW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,kWAA+WE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,KAAkBA,EAAE,SAAS,CAAC,SAAS,4CAA4C,CAAC,EAAE,KAAkBA,EAAE,SAAS,CAAC,SAAS,mCAAmC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yMAAoM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yRAAyR,CAAC,EAAeA,EAAE,aAAa,CAAC,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,4PAAoQE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,aAAa,CAAC,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,8VAAsWE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0GAA0G,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,wHAAwH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,6GAA6G,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,sFAAsF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,qEAAqE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,gIAAgI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qDAAqD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4KAA4K,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8UAA8U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sSAAiS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uUAAuU,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAAsP,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,4HAA4H,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,6LAA6L,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,kTAAkT,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,aAAa,CAAC,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,8PAAsQE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,sBAAmCE,EAAEC,EAAE,CAAC,KAAK,uBAAuB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,0KAAqK,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iKAA8KE,EAAEC,EAAE,CAAC,KAAK,4BAA4B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,4KAA4K,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeQ,EAAuBZ,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,qPAAkQE,EAAEC,EAAE,CAAC,KAAK,gDAAgD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,4WAA4W,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,4KAAkK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gDAA6DE,EAAEC,EAAE,CAAC,KAAK,sCAAsC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAE,gQAA2P,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iRAA4Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mbAA8a,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+NAAkOE,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,OAAO,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qYAAqY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oeAAoe,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wBAAwB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2XAA2X,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kSAAkS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wEAAmE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wgBAAwgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qaAAqa,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+XAAuYE,EAAEC,EAAE,CAAC,KAAK,4CAA4C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAE,gEAAgE,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,gEAAwEE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,6JAA6J,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,wcAAmc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mTAA8S,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qDAAqD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6UAA6U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yIAAyI,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,+BAA+B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAE,+BAA+B,CAAC,CAAC,CAAC,CAAC,CAAC,EAC9v+IS,EAAqB,CAAC,QAAU,CAAC,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,SAAW,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,mBAAqB,CAAC,KAAO,UAAU,CAAC,CAAC",
  "names": ["richText", "u", "x", "p", "Link", "motion", "richText1", "richText2", "richText3", "richText4", "richText5", "richText6", "richText7", "richText8", "__FramerMetadata__"]
}