{
  "version": 3,
  "sources": ["ssg:https://framerusercontent.com/modules/YVgynkVt96a0ES9Gk8yz/e2OiwG21hMIpRxQ1FbTF/eo4RAmtig-31.js"],
  "sourcesContent": ["import{jsx as e,jsxs as t}from\"react/jsx-runtime\";import{Link as a}from\"framer\";import{motion as n}from\"framer-motion\";import*as i from\"react\";export const richText=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Machine learning stands for a specialized approach to training computers without the need for explicit programming. The crucial difference from conventional programming lies in the fact that the developer does not have to create highly structured code in order to instruct the system.\"}),/*#__PURE__*/e(\"p\",{children:\"Any kind of learning becomes easier when someone helps you, that is, supervises you. Likewise, in machine learning, a machine learns faster if there is something that guides it through the processes it needs to understand. But we should not forget that although supervised learning is considered to be the most extensively studied and the most common type of machine learning, there are several types of machine learning, which will be discussed in more detail below.\"}),/*#__PURE__*/e(\"p\",{children:\"But why train machines? Machine learning is applied in situations where humans do not know precisely how to design an algorithm to solve a problem. These are such tasks as recognizing traffic signs, reading handwritten text, and other tasks that people solve every day. Yet we do not clearly comprehend how we accomplish it.\"}),/*#__PURE__*/e(\"p\",{children:\"Machine learning algorithms are employed to handle such tasks in order to help humans simplify and often speed up their work. That is, unless the algorithm is clear, in which case developers just create specific software code.\"}),/*#__PURE__*/e(\"p\",{children:\"Machine learning utilizes data and algorithms to emulate human behavior and represents one of the areas of Artificial Intelligence. AI refers to an artificially created system that is equipped with the capability to replicate the intellectual and cognitive processes performed by human beings.\"}),/*#__PURE__*/e(\"p\",{children:\"Further, we will take a closer look at supervised and unsupervised machine learning, we will understand what is labeled and unlabeled data, why it is needed, and we will also briefly touch upon machine learning models.\"}),/*#__PURE__*/e(\"h2\",{children:\"Supervised and unsupervised learning\"}),/*#__PURE__*/e(\"p\",{children:\"Supervised learning (a concept closely related to active learning) and unsupervised learning. Both groups exist and are utilized for different purposes. The key distinguishing feature of a supervised learning algorithm lies in the fact that the machine studies with labeled training data. For algorithms of this type, we need a supervisor mechanism that will supervise these algorithms.\"}),/*#__PURE__*/e(\"p\",{children:'Meanwhile, in unsupervised machine learning, the machine becomes smarter with the help of unlabeled data. So there are no correct choices assigned to the data. The machine has to find dependencies between objects or information on its own. As the name suggests, these algorithms have to learn \"on their own\".'}),/*#__PURE__*/e(\"p\",{children:\"Labeled data stems out of the labeling process where certain objects in raw data, such as videos or images, are identified and labeled. It helps machine learning models make accurate predictions and estimates. Commonly, the data is labeled by humans. A machine learning model refers to a program or file that is taught to identify particular kinds of patterns.\"}),/*#__PURE__*/e(\"h2\",{children:\"Supervised learning\"}),/*#__PURE__*/e(\"p\",{children:\"Supervised learning is the most common and researched kind of machine learning since it is much easier to train a machine with labeled training data. With this type of machine learning, certain experts generate what is called a training dataset. They label data to produce a set of examples and correct answers from an expert's point of view.\"}),/*#__PURE__*/e(\"p\",{children:\"As an example, the set may represent pictures of highways, where all objects are labeled by a certain label. A dataset can also consist of videos, texts, or audio files, depending on the type of data a model has to work with.\"}),/*#__PURE__*/e(\"p\",{children:\"After labeling, the resulting dataset with labels is presented, or in other words, fed to the ML model by ML engineers. The purpose of all of this is to train the machine learning model to figure out what it will need to predict in the future from examples. Labeled training data provides information to supervise the model's predictions. Supervised learning may be employed for two kinds of algorithms, based on the type of prediction desired.\"}),/*#__PURE__*/e(\"h3\",{children:\"Classification\"}),/*#__PURE__*/e(\"p\",{children:\"The goal of classification involves separating objects according to a pre-defined feature. With the help or better said under the supervision of labeled training data, the machine learns to identify the properties in the unlabeled data, which will be provided to it after successful training. It is suitable for classifying articles by topic or language, images by their content, music by its genre, users by their preferences, and the letters in people's mailboxes.\"}),/*#__PURE__*/e(\"h3\",{children:\"Regression\"}),/*#__PURE__*/e(\"p\",{children:\"Regression is about predicting a number, not a trait or category as in classification tasks. The machine attempts to draw a line that reflects the average dependency. With regression, we may calculate the predicted cost of a car by its mileage, the chance of occurrence of traffic jams based on the time of day, the expected demand for a product, and so on.\"}),/*#__PURE__*/e(\"h2\",{children:\"Supervised machine learning: labeled data\"}),/*#__PURE__*/e(\"p\",{children:\"Obviously, we can see how computer systems have evolved over the past twenty to thirty years and what they are capable of, including progress in artificial intelligence. Some of their capabilities were unimaginable only a few years ago.\"}),/*#__PURE__*/e(\"p\",{children:\"However, without human help, AI-based computer systems are very limited and often require data labeling process in order to become useful.\"}),/*#__PURE__*/e(\"p\",{children:\"For large companies as well as for the average individual, data is the most useful resource. All human experience can be called data, which they accumulate throughout their lives. For many, their experiences have not come easy for them. Similarly, data annotation process may often be one of the most expensive and time-consuming expenditures on a project. Although seemingly trivial, labeling requires a huge amount of time and energy.\"}),/*#__PURE__*/e(\"p\",{children:\"Data labeling as a concept and a field of expertise has emerged due to the need to provide large volumes of specifically prepared data as input for training systems. In the process of labeling, a qualitative transformation is carried out - raw data is supplemented with labels and turned into labeled datasets.\"}),/*#__PURE__*/e(\"p\",{children:\"A labeled dataset consists of processed and structured information. In addition to the already mentioned labeled images, video, and sound files, datasets are often presented in a table format. The rows of such a table are called objects or data points, and the columns are called features. Altogether, they are the labeled set of data, which is the basis for machine learning.\"}),/*#__PURE__*/e(\"p\",{children:\"Labeling data is quite a routine and tedious process. For instance, in order to train an application to recognize plants from a photo, it is necessary to identify vegetation in hundreds or even thousands of images. By doing this, the ML model is presented with information it can use to be able to identify plants in any photo.\"}),/*#__PURE__*/e(\"p\",{children:\"The task is more complicated because an ML engineer responsible for this model has to plan which kind of data they need, how much of it, and what features it should have. And all of that before the actual labeling process starts, because you cannot simply train the model with random annotated data: to avoid biases and incorrect training, you need to plan it in advance.\"}),/*#__PURE__*/e(\"h2\",{children:\"Process of data labeling\"}),/*#__PURE__*/e(\"p\",{children:\"The role of labeling experts in a company dealing with manual labeling of data may be performed by specially hired specialists with narrow expertise. For example, in the medical field to recognize features on X-rays or traffic experts to label videos or photos for future autonomous cars software.\"}),/*#__PURE__*/e(\"p\",{children:\"Yet you don't have to have special knowledge to label data. Not all projects and content require expert knowledge. Often companies hire people on so-called crowdsourcing platforms, where a large number of contributors label a large amount of data needed for a company project in a relatively short amount of time. This is one of the fastest and most financially affordable ways to acquire a lot of data assigned with labels.\"}),/*#__PURE__*/e(\"p\",{children:\"However, to choose a team of responsible people who will execute their tasks properly, the contractor has to come up with a reliable QA system. This system will have to check the skills of the participants before they start their work on the project and afterward will check the final data for inaccuracies and errors.\"}),/*#__PURE__*/e(\"p\",{children:\"Also, companies or startups turn to qualified outsourcing organizations or to a crowdsourcing platform for data set creation or data labeling tasks, involving hundreds or thousands of employees who label the data. Then, to eliminate the possibility of errors, QA managers review and double-check the result of labeling.\"}),/*#__PURE__*/e(\"p\",{children:\"To accomplish a task as extensive as data labeling, labeling experts are aided by ML engineers or data scientists, and use special software. Some companies develop it from scratch to achieve maximum confidentiality, while others prefer to use ready-made solutions.\"}),/*#__PURE__*/e(\"p\",{children:\"In addition to the above approaches, it is possible to resort to a simpler approach and find a ready-to-use open-source dataset for your tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"The types of data that may require labeling can be very diverse. For instance, it may be a dataset consisting of audio files with voice recordings. To create software that can recognize human speech, such data has to be accompanied by a transcript. Further such a dataset has to be fed to an ML model. There should be a whole range of such fragments so that future software could work efficiently and accurately.\"}),/*#__PURE__*/e(\"p\",{children:\"A comparable example would be creating software for self-driving cars that will have to identify items on the road, such as pedestrians, road signs, markings, borders, and so on. The ML model training that would be at the core of such an application requires a massive number of labeled video and/or photo files containing all these objects with the appropriate labels.\"}),/*#__PURE__*/e(\"h2\",{children:\"Types of data labeling\"}),/*#__PURE__*/e(\"p\",{children:\"Various data annotation techniques have been proposed for a wide range of purposes. Here are some of the most popular types these days.\"}),/*#__PURE__*/e(\"h3\",{children:\"Data processing for computer vision models\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Object detection: identifying 2D and 3D objects, for example with bounding boxes.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Object classification: sorting images into categories based on their content.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Image segmentation: selecting objects in images.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Computer vision (CV) refers to the computer science branch which attempts to duplicate human eyesight and perception. In other words, computer vision tries to analyze images and videos to retrieve information from what it sees.\"}),/*#__PURE__*/e(\"p\",{children:\"A CV system trained on such data is employed to automatically perform image recognition, classify images, find similar images, locate objects, assist in navigation, identification, and so on.\"}),/*#__PURE__*/e(\"h3\",{children:\"Natural language processing (NLP)\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Text recognition: transcription of printed or handwritten text.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Named entity recognition and semantic segmentation: assigning labels to different parts of text.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Text classification.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis: analyzing the tone of texts.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Natural language processing, as well as CV, assists computers, except that instead of helping them to see the world around them, it helps computers comprehend and interpret human language.\"}),/*#__PURE__*/e(\"p\",{children:\"An NLP model (often a text recognition and/or classification model) can be employed to create voice assistants, analyze the mood of a speaker, perform optical character recognition, translation, and much more.\"}),/*#__PURE__*/e(\"h3\",{children:\"Audio processing\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Audio recognition: transcribing texts for audio files.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Assigning labels to sounds, etc.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Audio file labeling is the foundation for speech recognition technology and is employed in such fields as voice assistant development, voice-to-text transition, and so on.\"}),/*#__PURE__*/e(\"p\",{children:\"Data plays a major role in machine learning. There should be enough of it or even a lot of it. And remember that you need high data quality. It often happens that some projects are canceled or postponed indefinitely due to the fact that it is simply impossible to collect data.\"}),/*#__PURE__*/e(\"p\",{children:\"Supervised machine learning is simply impossible without labeled data. Previously, we have described how you can label the necessary data, but if these methods are not available to you in any way, you can use unlabeled data.\"}),/*#__PURE__*/e(\"h2\",{children:\"Unsupervised learning\"}),/*#__PURE__*/e(\"p\",{children:\"As we have already learned, supervised learning is best suited for problems where there is an impressive set of reliable labeled data to train the algorithm. But that is not always the case. Lack of data is the most common problem in machine learning. Therefore, unsupervised learning exists, a method of machine learning in which a model is trained to identify patterns and hidden dependencies in unlabeled data sets without any human oversight.\"}),/*#__PURE__*/e(\"p\",{children:\"Unsupervised learning is less frequently applied in reality. However, this type of ML should not be considered to be inferior or superior to supervised learning. It is just employed for different kinds of tasks. There are tasks or circumstances where there is simply no other choice. For example, if labeled data cannot be found for a project, or if the project budget does not allow for high-quality training data labeling.\"}),/*#__PURE__*/e(\"p\",{children:\"Despite this, unsupervised learning algorithms may be employed to solve more complex processing problems compared to supervised learning. At the same time, the result of this type of ML is frequently unexpected and there are often no obvious regularities. Since machines lack labels to learn from, the goal of unsupervised learning is to discover patterns in the data and aggregate them. It solves 3 types of problems:\"}),/*#__PURE__*/e(\"h3\",{children:\"Clustering\"}),/*#__PURE__*/e(\"p\",{children:\"The task of clustering consists in distinguishing items by unknown characteristics. Clustering is essentially a classification, but with no classes known in advance. The clusterization process aims to find similar objects and group similar elements into clusters.\"}),/*#__PURE__*/e(\"p\",{children:\"Some examples of clustering can be found in photo apps that identify people's faces in photos and group them into photo albums. The app doesn't even know people's names, although it can distinguish them by their facial features.\"}),/*#__PURE__*/e(\"h3\",{children:\"Association rule learning\"}),/*#__PURE__*/e(\"p\",{children:\"This ML method solves the problem of understanding the rules and meanings of different item groups. It is used for recommendations in online stores and for finding correlations between customer purchases. Retailers may learn which items were bought together and apply that information to increase their sales.\"}),/*#__PURE__*/e(\"p\",{children:\"Association rules allow us to find out when and under what circumstances customers purchase certain product combinations. Information regarding past shopping patterns and the timing of those purchases helps to establish a discount program and generate personalized offers to boost sales.\"}),/*#__PURE__*/e(\"h3\",{children:\"Dimensionality reduction\"}),/*#__PURE__*/e(\"p\",{children:\"This kind of method assembles particular features into higher-level abstractions. It entails a data transformation so that the amount of data is reduced. This technique is employed to remove uninformative and redundant data which complicates processing from the set. It reduces the size of memory that is required for the dataset and speeds up the work of ML algorithms.\"}),/*#__PURE__*/e(\"h2\",{children:\"Unlabeled data\"}),/*#__PURE__*/e(\"p\",{children:\"As mentioned earlier, data labeling requires time and sometimes the specialized knowledge of a skilled professional. For instance, it would take a huge amount of time for a person to label ten thousand photos of handwritten text to solve the problem of handwriting recognition.\"}),/*#__PURE__*/e(\"p\",{children:\"In contrast, unlabeled input data may be used in its original form, that is, in its unprepared state. That's why they are also called raw data.\"}),/*#__PURE__*/e(\"p\",{children:\"Besides the use of unlabeled data in unsupervised learning, it is also applied in semi-supervised learning. This is a type of supervised learning, which employs a small amount of labeled data and a comparatively larger amount of unlabeled data.\"}),/*#__PURE__*/e(\"p\",{children:\"Training a semi-supervised learning model starts with training on a small amount of data assigned with labels. After a thorough learning process, a large amount of unlabeled data is fed to the algorithm. The model should independently determine the labels for such unlabeled data, which will be referred to as pseudo-labels or synthetic labeling. Then the previously labeled set of data is blended with unlabeled data, which now has pseudo labels, so in fact, it has become labeled as well. The model can now be trained on such mixed data.\"}),/*#__PURE__*/e(\"p\",{children:\"Another subset of machine learning called reinforcement learning employs a very low amount of unlabeled data. Such algorithms do not require labels for learning, since they allow the computer to develop an optimal strategy on its own as it interacts with the surrounding environment.\"}),/*#__PURE__*/e(\"p\",{children:\"Reinforcement implies constant improvement of oneself, whereas learning implies continuous environmental perception and feedback from the environment. Reinforcement and learning represent a continuous and repetitive process until the system achieves the optimal solution to a problem that needs solving. A reward in reinforcement learning happens to be the most important thing, as it represents feedback that indicates how well the algorithm is working at the current moment in time.\"}),/*#__PURE__*/e(\"p\",{children:\"There is also the concept of transfer learning. It is a problem in machine learning that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem. For example, pre-trained models which can recognize cars could be applied when trying to recognize trucks.\"}),/*#__PURE__*/e(\"p\",{children:\"Thus, not all ML algorithms necessarily need labeled data to accomplish tasks. As we have discussed, unlabeled data may be used in unsupervised learning, semi-supervised learning, and reinforcement learning. Only its quantity will be different.\"}),/*#__PURE__*/e(\"p\",{children:\"However, it is worth remembering that in some cases unlabeled data may provide significant help and considerably increase the algorithm's precision, whereas in other cases it may even reduce the quality of the solution to the problem.\"}),/*#__PURE__*/e(\"p\",{children:\"To figure out which data you need to create ML models, you should define the goals of the project before embarking on it, and thoroughly examine the available data. Undoubtedly, using unlabeled data saves time and budget on data labeling. That's why unsupervised learning, semi-supervised learning, and reinforcement learning have become great solutions in many fields involving terabytes of unlabeled data.\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/e(\"p\",{children:\"Machine learning has undoubtedly been firmly embedded in our lives lately. It will only be evolving even faster and faster. Labeled data plays a huge role in the development of ML algorithms. Supervised machine learning is impossible without it, and it is the type of machine learning that is considered the most widespread and thoroughly studied. The computer is given a ready-made dataset, from which it learns how to make future predictions.\"}),/*#__PURE__*/e(\"p\",{children:\"Unsupervised learning on the other hand employs unlabeled data. Compared to supervised learning its task is defined less clearly. It is more challenging for a machine to solve such a task since it applies unlabeled data, which means that such a system does not possess a predetermined correct answer and should identify the patterns on its own.\"}),/*#__PURE__*/e(\"p\",{children:\"However, a well-thought-out course of action helps developers obtain the solutions they need with unsupervised learning algorithms. Unlabeled data is also widely applied in a form of supervised learning - semi-supervised learning and in reinforcement learning.\"}),/*#__PURE__*/e(\"p\",{children:\"In addition, models of supervised learning tend to be more precise than the unsupervised learning ones, although they require direct human interference and accurate data labeling. On the contrary, unsupervised models explore unlabeled data on their own. But they still require some degree of human input to help them verify the output.\"}),/*#__PURE__*/e(\"p\",{children:\"The use of unlabeled data may substantially save time and money for the company since the labeling process is very time-consuming. Nevertheless, the results of unsupervised learning, semi-supervised learning, and reinforcement learning algorithms should also be validated by humans to avoid errors and inaccuracies.\"}),/*#__PURE__*/e(\"p\",{children:\"Therefore, when choosing an approach to machine learning, everything depends on how we formulate the problem that needs to be solved for a particular project. Furthermore, many tasks are easily defined as one type of learning and then can be transformed into another.\"}),/*#__PURE__*/e(\"p\",{children:\"As the world changes dramatically, machine learning is becoming more and more integrated into our daily lives. Understanding the basics will help you navigate the world better and allow you to make better judgments about today's modern ML technologies. Humans have significantly expanded their capabilities with AI, and soon, these capabilities will become even greater.\"})]});export const richText1=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Machine learning models are constantly being improved to increase accuracy and robustness to noise \u2014 the ability to make correct predictions on noisy data. This is particularly crucial for image classification algorithms, since real-world images often contain noise from various sources, depending on the weather conditions, time of day, and other factors. \"}),/*#__PURE__*/e(\"p\",{children:\"To evaluate noise robustness, we conducted an experiment to see if human labelers or ML models are better at classifying increasingly noisy images. We chose the fine-tuned ResNet-50 model as our classification model. Our goal was to identify the acceptable noise levels for both the model and humans and test our hypothesis that people are better at classifying noisy images. The results showed some intriguing and unexpected findings. Read on to see which group handles noisy images better.\"}),/*#__PURE__*/e(\"h2\",{children:\"How to add noise to images\"}),/*#__PURE__*/e(\"p\",{children:\"Let's first look at the basic principles of image formation. Pictures are composed of small units called pixels, each with a value that represents its color and brightness. These values are represented by three components: red, green, and blue, each ranging from 0 to 255. The combination of these three numbers determines the final color of the pixel, with high values creating bright colors and low values creating dark colors. For instance, a pixel with red, green, and blue values of 255, 0, 0 is a purely red pixel, and 255, 255, 255 represents a white pixel. The arrangement of pixel values and components creates the overall appearance of an image.\"}),/*#__PURE__*/e(\"img\",{alt:\"How to add noise to images\",className:\"framer-image\",height:\"633\",src:\"https://framerusercontent.com/images/d0sPHlHabD4b2Drk1XTGwk2Yqg.webp\",srcSet:\"https://framerusercontent.com/images/d0sPHlHabD4b2Drk1XTGwk2Yqg.webp?scale-down-to=512 512w,https://framerusercontent.com/images/d0sPHlHabD4b2Drk1XTGwk2Yqg.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/d0sPHlHabD4b2Drk1XTGwk2Yqg.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/d0sPHlHabD4b2Drk1XTGwk2Yqg.webp 2560w\",style:{aspectRatio:\"2560 / 1267\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:\"Adding noise to images involves intentionally altering some pixel values in the image. This mimics real-world conditions like transmission errors, camera noise, etc. Two common types of noise that can be added are salt-and-pepper noise and Gaussian noise.\"}),/*#__PURE__*/e(\"p\",{children:'Salt-and-pepper noise is a type of image noise that randomly changes some pixels to either black (\"pepper\") or white (\"salt\"), at a frequency determined by a fixed probability value p. A larger value of p will result in more pixels being changed in the image. The proportion of black and white pixels is usually equal, but it can be adjusted.'}),/*#__PURE__*/e(\"img\",{alt:\"Image with added salt-and-pepper noise, p=0.1 (left) and p=0.4 (right)\",className:\"framer-image\",height:\"506\",src:\"https://framerusercontent.com/images/5a8n6vBQ4BcEt7avkpNVF1eYOI.webp\",srcSet:\"https://framerusercontent.com/images/5a8n6vBQ4BcEt7avkpNVF1eYOI.webp?scale-down-to=512 512w,https://framerusercontent.com/images/5a8n6vBQ4BcEt7avkpNVF1eYOI.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/5a8n6vBQ4BcEt7avkpNVF1eYOI.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/5a8n6vBQ4BcEt7avkpNVF1eYOI.webp 2560w\",style:{aspectRatio:\"2560 / 1013\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Image with added salt-and-pepper noise, p=0.1 (left) and p=0.4 (right)\"})}),/*#__PURE__*/e(\"p\",{children:\"On the other hand, Gaussian noise introduces random variations to each pixel\u2019s value based on a normal distribution. Random values, drawn from a normal distribution with an expected value of 0 and standard deviation \u03C3, are added to each RGB channel. The larger the value of \u03C3, the more the pixel values can be altered, making the image increasingly noisy.\"}),/*#__PURE__*/e(\"img\",{alt:\"Image with added Gaussian noise, \u03C3=10 (left) and \u03C3=30 (right)\",className:\"framer-image\",height:\"506\",src:\"https://framerusercontent.com/images/n4gEkmKDxZJG6VjTVTCY2a24s.webp\",srcSet:\"https://framerusercontent.com/images/n4gEkmKDxZJG6VjTVTCY2a24s.webp?scale-down-to=512 512w,https://framerusercontent.com/images/n4gEkmKDxZJG6VjTVTCY2a24s.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/n4gEkmKDxZJG6VjTVTCY2a24s.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/n4gEkmKDxZJG6VjTVTCY2a24s.webp 2560w\",style:{aspectRatio:\"2560 / 1013\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Image with added Gaussian noise, \u03C3=10 (left) and \u03C3=30 (right)\"})}),/*#__PURE__*/e(\"p\",{children:\"There are noticeable differences between the two types of noise. Salt-and-pepper noise is uniformly distributed throughout the image, while Gaussian noise tends to be more noticeable in the white and black parts of the image, particularly in areas with object contours. Gaussian noise makes white areas appear darker since it adds positive values to RGB values close to 255, 255, 255 (with a maximum value of 255). This causes overflow and results in a darker pixel. These differences play a critical role in the performance of ML models, which we\u2019ll cover in more detail later.\"}),/*#__PURE__*/e(\"h2\",{children:\"ResNet-50 vs. crowdsourcing\"}),/*#__PURE__*/t(\"p\",{children:[\"As mentioned, our goal was to identify and compare the limits of human labelers and ML models in terms of classifying images with different noise levels. We wanted to understand the impact of the level and type of noise in the training dataset on the performance of these two groups. To do this, we used a binary classification \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/datasets/Bingsu/Cat_and_Dog\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"dataset of cats and dogs\"})}),\" with balanced classes. We added salt-and-pepper noise with probability values ranging from 0.1 to 0.9 and Gaussian noise with standard deviations ranging from 20 to 80. In total, we obtained 16 datasets with different levels of noise, including the original without any changes.\"]}),/*#__PURE__*/e(\"h3\",{children:\"The model\"}),/*#__PURE__*/t(\"p\",{children:[\"We used the \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/microsoft/resnet-50\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"ResNet-50\"})}),\" model from Hugging Face pre-trained by Microsoft as our classification model. Its relative lightness, among other features, made it a good choice since we were only focused on conducting a binary classification and this model contained a huge amount of data in the pre-training dataset. We fine-tuned the model for 4 epochs on each of the 16 datasets, resulting in 16 different fine-tuned models. Then, we used each fine-tuned model to calculate the accuracy of the predictions on each of the 16 datasets, creating a matrix that showed how the noise in the training sample affects prediction quality on a dataset with different noise.\"]}),/*#__PURE__*/e(\"p\",{children:\"In addition, we combined all 16 datasets into one and fine-tuned the model on it for 1 epoch. This gave us a model with exposure to all types and levels of noise, which we expected to perform better than the others. Below, we compare this model with our crowdsourcing results.\"}),/*#__PURE__*/e(\"h3\",{children:\"Crowdsourcing\"}),/*#__PURE__*/t(\"p\",{children:[\"We used the Toloka platform to assess human performance. To ensure accuracy and avoid any misunderstanding of the task or potential fraud, we provided training to Tolokers and used quality control rules like majority vote, control tasks, and limiting fast responses, among others. We labeled all 16 datasets, with each image annotated by three different labelers. Finally, we used the \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/en/docs/crowd-kit/reference/crowdkit.aggregation.classification.dawid_skene.DawidSkene\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Dawid-Skene\"})}),\" model to aggregate the responses.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Results\"}),/*#__PURE__*/e(\"p\",{children:'The results of the model\\'s accuracy are displayed in the matrix below. Take a look at any cell. The label on the left indicates the dataset on which the model was fine-tuned, and the label at the bottom is the dataset on which predictions were made. The cell\\'s value shows the accuracy of the fine-tuned model on the specific test dataset. The bottom two rows, \"all_noises\" and \"crowd\", show the results of the ResNet-50 model fine-tuned on all 16 datasets and crowdsourcing results, respectively.'}),/*#__PURE__*/e(\"img\",{alt:\"Accuracy matrix comparing crowdsourcing and ResNet-50\",className:\"framer-image\",height:\"993\",src:\"https://framerusercontent.com/images/jq5RLWWbq6UQTJVfVqS91hle2ms.webp\",srcSet:\"https://framerusercontent.com/images/jq5RLWWbq6UQTJVfVqS91hle2ms.webp?scale-down-to=512 512w,https://framerusercontent.com/images/jq5RLWWbq6UQTJVfVqS91hle2ms.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/jq5RLWWbq6UQTJVfVqS91hle2ms.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/jq5RLWWbq6UQTJVfVqS91hle2ms.webp 2560w\",style:{aspectRatio:\"2560 / 1987\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Accuracy matrix comparing crowdsourcing and ResNet-50\"})}),/*#__PURE__*/e(\"p\",{children:\"From this figure we can clearly see that the model\u2019s accuracy is lower when it is trained on one noise type and evaluated on another. This is especially evident when comparing performance on salt-and-pepper noise and Gaussian noise. For instance, if a model is trained on salt-and-pepper noise with p=0.3, it will perform well on salt-and-pepper noise, but give random predictions on images with Gaussian noise with std=10. This is because Gaussian noise differs from salt-and-pepper, as we explained above. Additionally, the first column reveals that the model's performance declines when making predictions on a clean dataset after being fine-tuned on a noisy dataset.\"}),/*#__PURE__*/e(\"h3\",{children:\"Humans vs machines\"}),/*#__PURE__*/e(\"p\",{children:\"To compare human and model performance, we created another diagram that highlights the two bottom rows of the matrix.\"}),/*#__PURE__*/e(\"img\",{alt:\"Accuracy of crowdsourcing and ResNet-50 fine-tuned on all noise levels\",className:\"framer-image\",height:\"563\",src:\"https://framerusercontent.com/images/rASCzFDVPsZTOh5VpQCRealDJQ.webp\",srcSet:\"https://framerusercontent.com/images/rASCzFDVPsZTOh5VpQCRealDJQ.webp?scale-down-to=512 512w,https://framerusercontent.com/images/rASCzFDVPsZTOh5VpQCRealDJQ.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/rASCzFDVPsZTOh5VpQCRealDJQ.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/rASCzFDVPsZTOh5VpQCRealDJQ.webp 2560w\",style:{aspectRatio:\"2560 / 1127\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Accuracy of crowdsourcing and ResNet-50 fine-tuned on all noise levels\"})}),/*#__PURE__*/e(\"p\",{children:\"The two graphs show that humans are outperformed on strong noise by the ResNet-50 model, which was trained on all types of noise.\"}),/*#__PURE__*/e(\"p\",{children:\"At first, the crowdsourced labelers had nearly perfect accuracy on weak noise, but their performance declined significantly compared to the ML model once salt-and-pepper noise with p=0.4 and Gaussian noise with \u03C3=60 was introduced. Surprisingly, the model was more robust to noise than humans.\"}),/*#__PURE__*/e(\"h3\",{children:\"How the crowd falls short\"}),/*#__PURE__*/e(\"p\",{children:\"To understand why the individual humans made mistakes, we looked at the images that were mislabeled by every person (meaning no one answered correctly).\"}),/*#__PURE__*/e(\"p\",{children:\"Can you tell if it's a cat or a dog in the images below? We couldn't either.\"}),/*#__PURE__*/e(\"img\",{alt:\"Example of image with salt-and-pepper noise with p=0.6 (cat)\",className:\"framer-image\",height:\"631\",src:\"https://framerusercontent.com/images/S34LPAvB2Edl2sy6sW2NIgqibQ.webp\",srcSet:\"https://framerusercontent.com/images/S34LPAvB2Edl2sy6sW2NIgqibQ.webp?scale-down-to=512 512w,https://framerusercontent.com/images/S34LPAvB2Edl2sy6sW2NIgqibQ.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/S34LPAvB2Edl2sy6sW2NIgqibQ.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/S34LPAvB2Edl2sy6sW2NIgqibQ.webp 2560w\",style:{aspectRatio:\"2560 / 1263\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Example of image with salt-and-pepper noise with p=0.6 (cat)\"})}),/*#__PURE__*/e(\"img\",{alt:\"Example of image with Gaussian noise with std=60 (dog)\",className:\"framer-image\",height:\"631\",src:\"https://framerusercontent.com/images/wfxjQkgxMzAFn7V1xtE3svkbY.webp\",srcSet:\"https://framerusercontent.com/images/wfxjQkgxMzAFn7V1xtE3svkbY.webp?scale-down-to=512 512w,https://framerusercontent.com/images/wfxjQkgxMzAFn7V1xtE3svkbY.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/wfxjQkgxMzAFn7V1xtE3svkbY.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/wfxjQkgxMzAFn7V1xtE3svkbY.webp 2560w\",style:{aspectRatio:\"2560 / 1263\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Example of image with Gaussian noise with std=60 (dog)\"})}),/*#__PURE__*/e(\"p\",{children:\"We concluded that when the noise level is low, humans outperform the ResNet-50 model trained on all noises. However, as the noise level increases, the accuracy of humans drops from 100% on clean data to 81% for the strongest salt-and-pepper noise and 73% on the strongest Gaussian noise. In contrast, the accuracy of the model trained on all types of noise only drops slightly: from 97% to 94% and 81% for the strongest salt-and-pepper and Gaussian noises, respectively.\"}),/*#__PURE__*/e(\"h3\",{children:\"Making sense of the results\"}),/*#__PURE__*/e(\"p\",{children:\"To understand why the model performs better with higher levels of noise, we would need to apply different techniques to interpret the model. A simple explanation could be the vast number of images (14 million) in the ImageNet dataset, which was used to pre-train the ResNet-50 model. We can also assume that fine-tuning the model with various types and levels of noise allowed it to learn how to recognize patterns on noisy images \u2014 since it was trained on the same dataset 16 times under different noise conditions.\"}),/*#__PURE__*/e(\"h2\",{children:\"What\u2019s next?\"}),/*#__PURE__*/e(\"p\",{children:\"Having considered random noise, we can move on to adversarial attacks \u2014 an image change that can be used to fool computer vision and ML models. These attacks make small, intentional changes to an image that can\u2019t be detected by the human eye and are designed to cause a model to make incorrect predictions. Adversarial attacks are a real problem in high-stakes areas like medicine or security where incorrect predictions can have serious consequences.\"}),/*#__PURE__*/e(\"p\",{children:\"To understand how adversarial attacks work, imagine a scenario where you show an ML model a picture of a panda. The model can accurately recognize the panda and label it. However, by making small changes to the picture, like adding noise or changing certain pixels, it's possible to create an image that the model now incorrectly labels as a gibbon, for example.\"}),/*#__PURE__*/e(\"img\",{alt:\" it's possible to create an image that the model now incorrectly labels as a gibbon, for example.\",className:\"framer-image\",height:\"593\",src:\"https://framerusercontent.com/images/QUe4El598BL8W9gZj7Zf4V9GIn4.webp\",srcSet:\"https://framerusercontent.com/images/QUe4El598BL8W9gZj7Zf4V9GIn4.webp?scale-down-to=512 512w,https://framerusercontent.com/images/QUe4El598BL8W9gZj7Zf4V9GIn4.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/QUe4El598BL8W9gZj7Zf4V9GIn4.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/QUe4El598BL8W9gZj7Zf4V9GIn4.webp 2560w\",style:{aspectRatio:\"2560 / 1187\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:\"This is the basic idea behind an adversarial attack: to trick the model into making a wrong prediction by making unnoticeable changes to the input. \"}),/*#__PURE__*/e(\"p\",{children:\"It\u2019s crucial to understand how these attacks work and how to defend against them. Some approaches include training models on adversarial examples to make them more robust, or using techniques such as image denoising to remove adversarial perturbations \u2014 a change to a physical object that is deliberately designed to trick an ML system into mistaking it for something else. As the use of ML models continues to grow, it's important to stay alert about the threat of adversarial attacks and work to reduce their impact.\"}),/*#__PURE__*/e(\"p\",{children:\"One way to help ML models cope with adversarial attacks is to involve crowdsourcing. Humans can identify patterns and contextual information that a model may miss and provide input that improves the model's accuracy and robustness. This approach, known as human-in-the-loop, can help mitigate the impact of adversarial attacks on machine learning models. We\u2019ll go into more detail on this in one of our next posts. Stay tuned.\"})]});export const richText2=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"A few years ago, data and big data became buzzwords. Far less clear was how to process that data, making it hard to predict how much insight could be gleaned from it.\"}),/*#__PURE__*/e(\"p\",{children:\"Things have changed dramatically since then. Think about how often companies' customer service or products are mentioned online \u2013 social media, news articles, blog posts, online reviews, forums, internal and external emails, product marketing collateral, public relations content, presentations and other documents. That information can then be mined for business purposes and text analytics, taking decision-making to the next level.\"}),/*#__PURE__*/e(\"p\",{children:\"But this is where the problem starts. While a lot of data is produced and downloaded every day, the majority of it is unstructured. It doesn't fit into clearly defined frameworks, which makes it hard to process and get useful insights.\"}),/*#__PURE__*/e(\"p\",{children:\"The good news is that there are now tools like text mining and sentiment analysis that make it possible to automate the process of collecting, organizing, and analyzing massive amounts of business-related data. In this article, we will talk about text mining techniques, sentiment analysis types, why they are important, and how to apply them to gain actionable business insights.\"}),/*#__PURE__*/e(\"h2\",{children:\"Text mining vs. sentiment analysis: why do we need both?\"}),/*#__PURE__*/e(\"p\",{children:\"If we deal with unstructured data, it first has to be structured before businesses can use it. Text mining, or text data mining, is the process of extracting relevant information and actionable insights from text to help businesses make better decisions.\"}),/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis, also known as opinion mining, uses machine learning (ML) and natural language processing (NLP) to categorize text information as positive, negative, neutral, or beyond.\"}),/*#__PURE__*/e(\"p\",{children:\"Customers who visit a business's website to learn more about its offerings or make purchases leave a trail behind them with every click they make. Since that information can help determine the success or failure of business moves, assist in developing a data-driven strategy, improve the user experience, and more, it would be foolish to ignore it. Let's break down the process.\"}),/*#__PURE__*/e(\"p\",{children:\"The first step is to simply collect the information. Once that's done, text mining can be used to sort through and categorize it. Only then should sentiment analysis be leveraged to delve into deeper meaning and nuance.\"}),/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis in text mining can thus be applied to massive pools of data to identify subjective information such as customers' opinions and feelings. That helps with product development, improving customer experience (CX), increasing operational efficiency, and simplifying data analysis in general.\"}),/*#__PURE__*/e(\"p\",{children:\"For example, customer sentiment in social media posts might include the following:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Positive\"}),' words like \"great customer service\" or \"convenient location\" represent positive sentiment, and add to the positive sentiment score.']}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Negative\"}),' words like \"awful\", \"unfriendly\" or \"unwelcoming\" in customer data add to negative sentiment scores. Sarcasm will also fall into this category, although analyzing it is a much more complicated task in text analytics.']}),/*#__PURE__*/t(\"p\",{children:['Sometimes, words like \"average\" or \"fine\" are counted as ',/*#__PURE__*/e(\"strong\",{children:\"neutral\"}),\" words and add to neutral sentiment. This happens when a more complicated scale is used, not just the binary one consisting only of positive or negative scores.\"]}),/*#__PURE__*/e(\"p\",{children:\"When performing sentiment analysis, developers usually create their own sentiment lexicon which contains information about which words and phrases are considered positive or negative. Of course, this is an oversimplification, and in ML-based solutions, the logic used under the hood is usually much more complicated than that. ML models are trained on massive amounts of carefully selected training data, so that they are able to perform complicated tasks like key phrase extraction, feature extraction, analyzing unstructured text, taking into account word frequency and defining overall sentiment to provide valuable insights.\"}),/*#__PURE__*/e(\"p\",{children:\"An excellent illustration is when a company uses sentiment analysis on customer feedback to learn more about what their existing or potential customers think of their products and services. They might use surveys, emails, social media, news sites, or feeds. We'll take a closer look at some cases later.\"}),/*#__PURE__*/e(\"h3\",{children:\"What about aspect based sentiment analysis?\"}),/*#__PURE__*/e(\"p\",{children:\"There's also a special kind of sentiment analysis called aspect-based sentiment analysis. It allows businesses to analyze customer feedback not strictly based on the sentiment, but rather by different aspects of a product or service. Here's the difference:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Sentiments represent what customers feel, i.e. positive, neutral or negative opinions (sentiment scores).\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Aspects represent specific categories, features, or topics mentioned by customers.\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"What technologies are at the heart of text mining and sentiment analysis?\"}),/*#__PURE__*/e(\"p\",{children:\"In this section, we will look at the core technologies behind sentiment analysis in text mining and learn more about how they work.\"}),/*#__PURE__*/e(\"h3\",{children:\"Artificial intelligence (AI) and machine learning\"}),/*#__PURE__*/e(\"p\",{children:\"ML is a subfield of AI and computer science that simulates human learning. It uses data and algorithms without explicit programming to improve performance on a set of tasks. With that in mind, the heart of any AI solution is an ML model, or training algorithm, that is essentially a set of coded instructions from ML engineers telling the solution how to train itself.\"}),/*#__PURE__*/e(\"h3\",{children:\"NLP\"}),/*#__PURE__*/e(\"p\",{children:\"The subfield of ML and AI that allows computers to understand text and spoken words in the same manner as humans is known as NLP. It allows computers to process text or voice data with clarity by combining computational linguistics with statistical, machine learning, and deep learning models.\"}),/*#__PURE__*/e(\"p\",{children:\"Until about five years ago, language models in ML couldn't understand meaning in context, or how different words in the same sentence relate to one another. They could provide a solid foundation for sentiment analysis models, but they were not task-specific. Now, in order for an AI application to carry out specific NLP tasks such as sentiment analysis, it needs further training on task-specific data. Let's see how that works.\"}),/*#__PURE__*/e(\"h2\",{children:\"How to perform sentiment analysis\"}),/*#__PURE__*/e(\"p\",{children:\"Natural language processing algorithms require annotated datasets that serve as information-packed textbooks with practical examples and definitions. But that means ML engineers have to feed the model a large amount of labeled data, training it to perform text processing and draw conclusions about relationships between words and their connotations. Typical tasks in text-based data labeling might include language detection, part of speech tagging, named entity recognition, and so on.\"}),/*#__PURE__*/e(\"p\",{children:\"This layer of annotated datasets is used to fine-tune the model so it can accurately predict sentiment labels \u2013 words and phrases that carry particular ideas, feelings, or topic-specific messages. The datasets can be obtained in a variety of ways, including crowdsourcing on platforms such as Toloka.\"}),/*#__PURE__*/e(\"p\",{children:\"After this phase of training is complete, NLP tasks like sentiment analysis can be specifically tailored to the AI solution's language model. Sentiment analysis on text data is a type of ML-based NLP tool. It was trained to analyze texts for their emotional and other polarities, from positive to negative, meaning it could detect them autonomously.\"}),/*#__PURE__*/e(\"p\",{children:\"As a result, models used for sentiment analysis can be taught to go beyond literal meanings by considering nuances such as tone, sarcasm, and misuse.\"}),/*#__PURE__*/e(\"h2\",{children:\"How can text data benefit businesses?\"}),/*#__PURE__*/e(\"p\",{children:\"There are undoubtedly more beneficial ways to use sentiment analysis to gain data-driven insights for businesses than what we will cover in this article. We do, however, hope that the examples we provide will help you better understand what sentiment analysis and text mining are capable of.\"}),/*#__PURE__*/e(\"h3\",{children:\"Moderating content\"}),/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis performed on Facebook, YouTube, Twitter, and SoundCloud, as well as in more niche communities for industry professionals, helps identify posts that don't adhere to the platform's guidelines. As this is an automated process, it can be a huge time saver for moderators who would otherwise have to read through large volumes of text data.\"}),/*#__PURE__*/e(\"h3\",{children:\"Monitoring brand reputation\"}),/*#__PURE__*/e(\"p\",{children:\"Brand monitoring is similar to content moderation in many ways. Instead of simply blocking inappropriate or irrelevant information, however, businesses use sentiment analysis to focus on how customers feel about their brand and track changes in perception over time. They do this by parsing customer reviews for both positive and negative sentiments in order to better understand what customers like and dislike.\"}),/*#__PURE__*/e(\"p\",{children:\"You can also use comparative brand analysis to keep tabs on industry developments. Another option is to learn whether a given advertising campaign is yielding positive results or how consumers are responding to a newly released product.\"}),/*#__PURE__*/e(\"h3\",{children:\"Gaining insights from customer reviews\"}),/*#__PURE__*/e(\"p\",{children:\"This is also related to brand reputation. With that said, it focuses solely on customer feedback and reviews derived from survey responses, customer service data, blogs, social media, and elsewhere.\"}),/*#__PURE__*/e(\"p\",{children:\"With sentiment analysis capable of processing thousands of pieces of feedback in just minutes much more accurately than humans ever could, companies can identify areas for growth. They can enhance existing offerings and even reward customers for their feedback, both positive and negative.\"}),/*#__PURE__*/e(\"h3\",{children:\"Recommendation systems and targeted ads\"}),/*#__PURE__*/e(\"p\",{children:\"Recommendation systems are basically suggestion filters that e-commerce stores or online marketplaces such as Amazon or eBay rely on every day. How do they work? Sentiment analysis algorithms analyze users' explicit and implicit product preferences to learn about their tastes, driving timely, relevant product suggestions. In other words, the right offer at the right time.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:'\"We had two goals: get high-quality data we could use to train the recommendation system for our e-commerce platform and measure the accuracy of our current recommendation algorithm. Toloka helped us improve our model with super-fast labeling of tens of thousands of products from our store. Toloka makes the data problem easier so we can focus on our algorithms\".'})}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"\u2013 Ivan Lapitsky, Project Manager, Yandex Market\"})}),/*#__PURE__*/e(\"h3\",{children:\"Prioritizing customer issues in tickets and queries\"}),/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis is also commonly used in customer service and support. Incoming chatbots, emails, phone calls, and web queries are analyzed, sorted according to topic and urgency, and forwarded to the appropriate department or employee.\"}),/*#__PURE__*/e(\"p\",{children:\"Staff members don't have to manually search through query pools, while sentiment analysis still ensures that the most urgent problems are resolved right away.\"}),/*#__PURE__*/e(\"h3\",{children:\"Conducting market research\"}),/*#__PURE__*/e(\"p\",{children:\"Before releasing a product or launching a service, businesses usually conduct extensive market research to determine if there is sufficiently strong demand for their proposed offering. This is where sentiment analysis comes in handy.\"}),/*#__PURE__*/e(\"p\",{children:\"By collecting data on customer feedback about competing products and services, companies can gain valuable insight into their target market's pain points and unfulfilled needs. Their product offerings are then hyper-relevant to their target market.\"}),/*#__PURE__*/e(\"h2\",{children:\"Wrapping up\"}),/*#__PURE__*/e(\"p\",{children:\"Data collection and analysis can be daunting tasks, but they are essential if a company hopes to maintain its position as a market leader.\"}),/*#__PURE__*/e(\"p\",{children:\"As we've seen, there's a powerful solution: text mining combined with sentiment analysis. Data analysis leads to dynamic insights regardless of the type of text you need to analyze and without the months-long slog of manual review.\"})]});export const richText3=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"As we\u2019ve discussed in many of our previous posts in this blog, both data collection and data labeling (aka data annotation) play an essential role in Machine Learning (ML) and AI application development, as well as some more traditional domains not powered by AI (e.g., stock trading, linguistics, and archeology).\"}),/*#__PURE__*/e(\"p\",{children:\"At Toloka, our chosen data collection and data-labeling methodology is known as crowdsourcing, which is a type of global-crowd-powered outsourcing on a large scale. Apart from data collection and labeling for machine learning model training, Toloka\u2019s data annotators (aka crowd contributors aka \u201CTolokers\u201D) also collect and annotate data for machine learning model evaluation after training and machine learning model monitoring after deployment. To rephrase that, we help AI developers to make sure that their AI products do exactly what they\u2019re supposed to do right before and after the products are released.\"}),/*#__PURE__*/e(\"h2\",{children:\"Data collection and labeling for machine learning\"}),/*#__PURE__*/e(\"p\",{children:\"There are a number of common data collection and data-labeling applications that Toloka\u2019s clients (aka \u201Crequesters\u201D) are involved with. They include:\"}),/*#__PURE__*/e(\"h4\",{children:\"Natural Language Processing (NLP)\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Text classification\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Text summarization\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Named entity recognition (NER)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Machine translation\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Speech recognition\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Speech synthesis\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Question answering\"})})]}),/*#__PURE__*/e(\"h4\",{children:\"Social Media\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Content moderation\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Hate speech detection\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Opinion mining\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Brand reputation management\"})})]}),/*#__PURE__*/e(\"h4\",{children:\"E-commerce\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Product categorization / search-and-filter / rating systems\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Product recommendation / recommender systems\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Customer segmentation\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Price optimization\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Inventory management and forecasting\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Supply chain optimization\"})})]}),/*#__PURE__*/e(\"h4\",{children:\"Healthcare\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Medical image analysis (e.g., X-rays, CT scans, MRIs, etc)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Electronic health record (EHR) annotation and classification\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Disease diagnosis and prediction\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Drug discovery and development\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Patient monitoring and risk assessment\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Medical chatbots and virtual assistants\"})})]}),/*#__PURE__*/e(\"h4\",{children:\"Legal\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Contract analysis\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Legal document summarization\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Legal document classification\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Intellectual property analysis\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Case outcome prediction\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Legal research and analytics\"})})]}),/*#__PURE__*/e(\"h4\",{children:\"Cybersecurity\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Intrusion detection\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Malware classification\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Threat intelligence\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Video surveillance\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Access control\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"Data collection and labeling for computer vision\"}),/*#__PURE__*/e(\"p\",{children:\"One of the most ubiquitous (and cutting-edge) ML applications can be seen in the domain known as Computer Vision (CV). The term is quite self-explanatory \u2013 computer vision essentially enables machines to \u201Csee\u201D and recognize the world around them in order to execute a set of predetermined functions. Among some of the most common computer vision tasks, including those performed by Tolokers on our platform, are:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Image classification\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Image segmentation\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Image transcription\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Object detection (including bounding boxes and polygon)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Optical character recognition (OCR)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Facial recognition (including keypoint)\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Image recognition technology (i.e., knowing what a digital image shows more generally) and object recognition technology (i.e., the ability to identify objects in digital images) are some of the most popular CV tasks that have multiple real-world uses.\"}),/*#__PURE__*/e(\"p\",{children:\"At this point, we won\u2019t be going into the details of a typical image recognition pipeline, that is, how the whole process is organized and carried out from start to finish, beginning with a foundation model and ending with a ready downstream application. If you want to know more about this topic, we invite you to have a look at this blog post. The present article will instead briefly cover some of the most common image recognition methodologies and then focus on the more practical aspects and specific use cases of image recognition and object recognition technology.\"}),/*#__PURE__*/e(\"p\",{children:\"Also note that when we talk about image recognition, we don\u2019t necessarily mean that there\u2019s an actual robotic camera with a lens that\u2019s physically \u201Clooking\u201D at something (though this is one possibility, of course). Rather, in this context, image recognition refers to any scenario in which a system can accurately process digital images, understand them, and successfully fulfill its intended function.\"}),/*#__PURE__*/e(\"h2\",{children:\"What is under the hood of the image recognition technology\"}),/*#__PURE__*/e(\"p\",{children:\"Among some of the most well-known methodologies of image recognition using deep learning techniques and deep neural networks today are the following,. We\u2019re listing them here along with their main strengths and shortcomings:\"}),/*#__PURE__*/e(\"h3\",{children:\"Convolutional Neural Networks (CNNs)\"}),/*#__PURE__*/e(\"p\",{children:\"This is by far the most popular type of neural network for pretrained image recognition algorithms. It works by learning patterns and features in a digital image through convolution, i.e., \u201Cpooling layers.\u201D CNNs are good at recognizing complex patterns and visual objects regardless of their position, which is great for most image identification systems.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": High accuracy, ability to recognize complex patterns.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": Can be computationally expensive to train, i.e., buying and then running high-end hardware, namely Central Processing Units (CPUs) and Graphics Processing Units (GPUs), which can be costly both time- and money-wise.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Recurrent Neural Networks (RNNs)\"}),/*#__PURE__*/e(\"p\",{children:\"This type of network is designed to process image data sequentially (that is, bit by bit) and capture long-term dependencies and relationships, which is useful for video frames and other image recognition applications.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": Good at processing sequential data.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": Can also be computationally expensive.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Support Vector Machines (SVMs)\"}),/*#__PURE__*/e(\"p\",{children:\"These are linear models that find the \u201Chyperplane\u201D (i.e., a line between data points) in a high-dimensional space in order to separate different object classes within images.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": Simpler than neural networks and more computationally efficient.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": May not perform as well on more complex image recognition and object detection tasks.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Deep Belief Networks (DBNs)\"}),/*#__PURE__*/e(\"p\",{children:\"These are a type of deep learning image recognition network that\u2019s composed of multiple layers of \u201Chidden units\u201D that learn hierarchical representations used in image processing. This means that these image recognition algorithms can identify which patterns are more and which are less significant.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": Good at hierarchical learning and feature extraction.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": Can also be expensive to train (CPUs, GPUs, etc).\"]}),/*#__PURE__*/e(\"h3\",{children:\"K-Nearest Neighbors (KNNs)\"}),/*#__PURE__*/e(\"p\",{children:\"This approach uses a non-parametric method (i.e., it doesn\u2019t assume any particular distribution shape/type) in order to find the nearest training examples to a new image and then classify that image based on its nearest \u201Cneighbors.\u201D\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": Simple and easy to implement.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": Can be computationally expensive with larger datasets.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Transfer Learning models\"}),/*#__PURE__*/e(\"p\",{children:\"These models for image recognition work on large datasets for a particular task and subsequently fine-tuned for a different task using smaller datasets.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": Useful for more specialized image recognition and object identification tasks.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": May not perform well on all target tasks (i.e., may require lengthy fine-tuning periods in order to achieve optimal performance).\"]}),/*#__PURE__*/e(\"h3\",{children:\"Autoencoders\"}),/*#__PURE__*/e(\"p\",{children:\"Autoencoders work by learning a condensed version of image data (\u201Cencoding\u201D) and then proceed with reconstructing the original data (\u201Cdecoding\u201D). This can be especially useful for \u201Cdenoising,\u201D that is, cleaning \u201Cnoisy\u201D (corrupted or distorted) images during object detection tasks in various image identification systems.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": Good for denoising and compression.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": May not perform as well as other models on image classification tasks.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Faster R-CNN, Mask R-CNN, You Only Look Once (YOLO), and Single Shot MultiBox Detector (SSD)\"}),/*#__PURE__*/e(\"p\",{children:\"These are all specific implementations of object detection and image recognition models based on CNNs:\"}),/*#__PURE__*/e(\"h4\",{children:\"Faster R-CNN\"}),/*#__PURE__*/e(\"p\",{children:\"This type of networks uses a region proposal network to suggest object locations and a classification network to predict class labels, i.e., it basically suggests where the objects of interest are and what they are.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": High accuracy for object detection tasks, can be easily fine-tuned for specific use cases.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": Requires a large amount of training data, computationally expensive.\"]}),/*#__PURE__*/e(\"h4\",{children:\"Mask R-CNN\"}),/*#__PURE__*/e(\"p\",{children:\"It is a modified version of Faster R-CNN that predicts not only the location and class of an object in image recognition tasks, but also its precise shape or boundary as a set of pixels (called a \u201Cmask\u201D).\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": Predicts both the location and shape of objects in image recognition tasks, high accuracy for object detection tasks.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": Requires a large amount of training data, also computationally expensive.\"]}),/*#__PURE__*/e(\"h4\",{children:\"You Only Look Once (YOLO)\"}),/*#__PURE__*/e(\"p\",{children:\"This is a very popular one-stage model (not least among our requesters) that predicts class labels and bounding boxes of objects in an image. In other words, it can draw an outline of each object to show where it\u2019s located and tell us what kind of object it is exactly (e.g., a traffic cone, a person, a tree, etc).\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": Fast inference time (i.e., rapid responses), low memory usage.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": Lower accuracy compared to other models such as Faster R-CNN, may have difficulty detecting small objects.\"]}),/*#__PURE__*/e(\"h4\",{children:\"Single Shot MultiBox Detector (SSD)\"}),/*#__PURE__*/e(\"p\",{children:\"This is a one-stage model similar to YOLO, as it also identifies objects and informs us of their location. The difference is that SSD uses different scales to detect objects, which can improve accuracy but may also make image identification systems that rely on it run slower.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pros\"}),\": Offers a good balance between accuracy and performance speed, can detect objects of different sizes.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cons\"}),\": May struggle with identifying small objects, can be computationally expensive for larger models.\"]}),/*#__PURE__*/e(\"p\",{children:\"To sum up, choosing the best image recognition model depends on a number of factors, including the size of your dataset and the complexity of the objects being recognized. In other words, it all depends on the specific requirements of your task at hand. It\u2019s also crucial to carefully consider the trade-off between accuracy and computational resources when selecting a methodology: for better or worse, there\u2019s simply no magic pill here.\"}),/*#__PURE__*/e(\"p\",{children:\"In most cases, at least some computational expense is unavoidable, which is why optimizing your ML pipeline and having a robust work environment is essential, as is reducing your data-associated costs, including those for data collection and data labeling. With that in mind, let\u2019s now look at how this image recognition and object detection technology is being utilized by some of our requesters.\"}),/*#__PURE__*/e(\"h2\",{children:\"What is image recognition used for\"}),/*#__PURE__*/e(\"h3\",{children:\"Product development and moderation\"}),/*#__PURE__*/e(\"h4\",{children:\"Case 1\"}),/*#__PURE__*/e(\"p\",{children:\"AliExpress, a global e-commerce giant, faced a significant challenge of having to comply with the constantly changing and differing regulations in each of the ten CIS (Commonwealth of Independent States) countries it serves. To address this challenge, the company needed a highly scalable and efficient moderation system that would filter out illegal, culturally inappropriate, or inferior products from its platform.\"}),/*#__PURE__*/e(\"p\",{children:\"The company partnered with Toloka with the goal of creating one such system that would take into account each country\u2019s local laws and cultural differences. The first step was defining the five categories of products to be moderated, ranging from items that can be sold without restrictions to products that are prohibited by law.\"}),/*#__PURE__*/e(\"p\",{children:\"To handle the large volume of products on the platform, AliExpress prioritized items for moderation based on the frequency of views. These selected items were sent to a pre-trained ML model that predicted whether each item was acceptable or not, providing a confidence score to indicate the reliability of the verdict. Items with a high confidence score were added directly to the database, while uncertain ones were sent to Tolokers for human-handled verification.\"}),/*#__PURE__*/e(\"p\",{children:\"The annotation process consisted of three stages, with all crowd contributors taking an exam before getting assigned to the final task. Inconsistencies in answers were sent to a second task level, where response accuracy was rechecked.\"}),/*#__PURE__*/e(\"p\",{children:\"The platform\u2019s partnership with Toloka (i.e., ML pipeline optimization with human verification) reduced the price of verified items by half and improved their image recognition system and specifically, the moderation efficiency by 500-fold: from only 200 items per day to 100,000 daily items. AliExpress has since been able to scale up and handle 100,000 items per day with a 98.7% label quality.\"}),/*#__PURE__*/e(\"h4\",{children:\"Case 2\"}),/*#__PURE__*/e(\"p\",{children:\"Neatsy is an innovative app that creates a digital model of feet (yes, you read it right). It features a 3D scanner that generates a 3D model of the customer\u2019s foot, which it then uses to suggest the most appropriate footwear.\"}),/*#__PURE__*/e(\"p\",{children:\"The app is powered by a sophisticated neural network that\u2019s designed to effectively isolate human feet from the ground. The app\u2019s 3D scanner needed over 50,000 labeled images in order to train itself. Acquiring this many labeled images of feet within a tight deadline was a daunting challenge. The company explored several options before deciding to partner with Toloka, seeking a robust and speedy solution capable of handling image labeling at scale.\"}),/*#__PURE__*/e(\"p\",{children:\"A special training pool was created for our crowd contributors, followed by a competency test for each Toloker before they could join the taskforce. Three weeks after that, the company received all 50,000 labeled images.\"}),/*#__PURE__*/e(\"p\",{children:\"These labeled images played a crucial role in retraining Neatsy\u2019s neural network, resulting in a significant improvement in segmentation quality as compared to the baseline \u2013 the app\u2019s 3D scanner increased its accuracy by 12%.\"}),/*#__PURE__*/e(\"h4\",{children:\"Case 3\"}),/*#__PURE__*/e(\"p\",{children:\"An up-and-coming smartphone company faced a challenge in developing an intuitive CV algorithm that could detect and recognize different hand gestures from a vast database of images. However, the small in-house team lacked the resources (and the time) required to quickly label and classify all of the images, leading to serious delays in the product\u2019s time to market, with the \u201Crunway\u201D beginning to run out (i.e., investment covering ongoing expenses).\"}),/*#__PURE__*/e(\"p\",{children:\"To overcome this hurdle, the company sought the assistance of our crowd contributors to scale up the labeling volume. In just under two days, Tolokers were able to deliver 2000 new labels with a high accuracy rate of 93.5%. As a result, the company was able to improve their image recognition software and deliver a more user-friendly experience/interface to their customers.\"}),/*#__PURE__*/e(\"h3\",{children:\"Human face and body recognition\"}),/*#__PURE__*/e(\"h4\",{children:\"Case 1\"}),/*#__PURE__*/e(\"p\",{children:\"A Japanese startup approached Toloka with a challenge of labeling human faces in 34,000 images from various TV shows. Unsurprisingly, the company was looking for a solution that was fast, affordable, and dependable.\"}),/*#__PURE__*/e(\"p\",{children:\"The project faced its first challenge when it came to defining what should be considered a human face. This wasn\u2019t clear since some of the images contained anime characters, drawings, computer-generated imagery, and humanoid androids. After some deliberation, it was determined that human faces would encompass all characters in the images except for animated figures and manga.\"}),/*#__PURE__*/e(\"p\",{children:\"To tackle this, Toloka assigned different pay rates for processing images of varying complexity and tasked all crowd contributors with learning and labeling images of increasing difficulty levels. Moderators, in turn, checked for quality control, ensuring that each image was correctly labeled. The entire task was completed in three stages \u2013 introduction and practice, labeling, and quality control \u2013 with a smooth learning curve that allowed our crowd contributors to deliver high-quality work on more complex images.\"}),/*#__PURE__*/e(\"p\",{children:\"Over a three-week period, Tolokers labeled and submitted images with 65,000 faces at a cost of approximately $0.015 per face. This was estimated to be 250% lower than any non-crowdsourcing solution available on the market at the time of project completion, while maintaining quality at or above the market average. Other methods like CVAT (Computer Vision Annotation Tool), which the startup team had previously considered, would have required more labor-induced expenses (i.e., highly trained specialists) and also more time.\"}),/*#__PURE__*/e(\"p\",{children:\"Utilizing the power of Toloka\u2019s global crowd made it possible to label tens of thousands of faces at a fraction of the expected cost by using a much larger pool of non-expert participants and aggregating their results instead, i.e., overlapping multiple submissions per image to reach higher accuracy levels.\"}),/*#__PURE__*/e(\"h4\",{children:\"Case 2\"}),/*#__PURE__*/e(\"p\",{children:\"IVI is a video streaming platform that offers over 20,000 movies and TV series to its users, with personalized recommendations as one of its key features. Personalization of title posters is an important aspect of IVI\u2019s marketing strategy. To achieve this, multiple posters need to be created for each title.\"}),/*#__PURE__*/e(\"p\",{children:\"However, the process of handcrafting posters is very time-consuming. To save time, IVI created a tool called Parker, which generates high-quality title previews and posters automatically. Though this approach saves time, it has a major drawback \u2013 Parker can often produce posters with awkward-looking facial expressions when caught mid-frame, making them largely unusable. With no viable solution in sight and a pressing deadline for new releases, IVI turned to Toloka.\"}),/*#__PURE__*/e(\"p\",{children:\"To reliably retrain Parker\u2019s algorithm, IVI needed about 72,000 labeled images. To meet this requirement, the company extracted high-quality facial images from 18 different films, with each actor having 10 to 100 images.\"}),/*#__PURE__*/e(\"p\",{children:\"The task of labeling this data involved five stages: decomposition, instructions, task interface, quality control, and aggregation. Tolokers were tasked with answering a binary classification question \u201CIs this a normal facial expression or not?\u201D in two phases. The first phase was to confirm that each image indeed contained a human face, while the second phase was to answer the main question.\"}),/*#__PURE__*/e(\"p\",{children:\"IVI leveraged Toloka\u2019s ready image classification template for all phases and subphases of the labeling task. Each page displayed 8-10 images and 3 radio buttons (i.e., graphical control elements that can be pressed), allowing Tolokers to use their keyboard and mouse for inputs. To minimize errors and exclude bots or unscrupulous annotators, the project had four quality control mechanisms in place, including training exam, hidden tests, overlap, and assurance tools (these are various techniques used to make sure that crowd performers are real people who are paying close attention).\"}),/*#__PURE__*/e(\"p\",{children:\"All of the submitted responses were aggregated to get the final results. The assignment took four days \u2013 11 times faster than the estimated in-house alternative. The accuracy rate fluctuated somewhat initially and subsequently climbed to and stayed at 90% toward the end of the project.\"}),/*#__PURE__*/e(\"p\",{children:\"This significant improvement to Parker\u2019s image recognition algorithm through \u201Chuman-in-the-loop\u201D data labeling (i.e., human-handled annotation) allowed the platform to identify and prioritize attractive-looking posters and previews over inferior ones.\"}),/*#__PURE__*/e(\"h4\",{children:\"Case 3\"}),/*#__PURE__*/e(\"p\",{children:\"When it comes to missing person cases, law enforcement agencies have a variety of tools and tactics at their disposal. One such tool that has become increasingly popular in recent years is the use of drones, or unmanned aircraft systems (UAS), for surveillance and search-and-rescue operations across the United States and other parts of the world.\"}),/*#__PURE__*/e(\"p\",{children:\"In Virginia, the Loudoun County Sheriff\u2019s Office purchased an Indago drone that was deployed for search within 2 minutes of the first call and proved more agile than a helicopter. The drone was able to enter narrower and harder-to-reach spaces closer to the ground, ultimately leading to the quick discovery of a missing person. Likewise, in Connecticut, the Hartford Police Department has been able to locate several missing persons across large areas along the Connecticut River since the inclusion of UAS devices into their search-and-rescue operations.\"}),/*#__PURE__*/e(\"p\",{children:\"However, simply having drones and accompanying navigation equipment is not always enough. With a large volume of raw and often noisy (i.e., inconclusive) drone imagery, finding a missing person can feel like finding a needle in a haystack. For this reason, post-surveillance data labeling \u2013 that is, processing search imagery after the fact \u2013 has become an essential component of drone surveillance and image recognition in missing person cases.\"}),/*#__PURE__*/e(\"p\",{children:\"Liza Alert, a non-profit organization that searches for missing persons across Eastern Europe and Central Asia, has partnered with Toloka to utilize the power of our global crowd to carry out post-surveillance data labeling. Liza Alert teams use drones to search for missing persons on-site, often in the densely forested wilderness or secluded rural areas, and the imagery is then passed on to Tolokers for further processing.\"}),/*#__PURE__*/e(\"p\",{children:\"Our crowd contributors label the data and identify areas where the missing person may be located, which helps with their rescue efforts. Simultaneously, this also provides new annotated data that can be used to train and fine-tune future ML-powered applications for search-and-rescue purposes, which is a combination of object detection and facial recognition technologies.\"}),/*#__PURE__*/e(\"p\",{children:\"As of 2021, Tolokers processed over 330,000 drone images collected by Liza Alert, with 49,000 of those images containing people. This led to over 200 search missions and ultimately saved the rescue organization 330 working days, which is a significant improvement.\"}),/*#__PURE__*/e(\"p\",{children:\"The use of surveillance drones and crowdsourced data labeling has proved to be a powerful combination in missing person cases. With the ability to collect and process large volumes of data quickly and efficiently, law enforcement agencies and rescue organizations are better equipped now than ever to locate missing persons and bring them to safety.\"}),/*#__PURE__*/e(\"h2\",{children:\"Concluding remarks\"}),/*#__PURE__*/e(\"p\",{children:\"As we\u2019ve seen, creating image recognition tools and object detection algorithms is a rather complex process that may rely on any number of CV models, depending on the specifics of a given AI downstream application. In most cases, it comes down to the trade-off between time and accuracy, as well as computational resources (mainly hardware-related) that are required to carry out ML training and fine-tuning.\"}),/*#__PURE__*/e(\"p\",{children:\"Because of this, it\u2019s important to have a well-structured ML pipeline in place along with a robust infrastructure that can facilitate work at every stage. It\u2019s equally important to possess the ability to transition from one stage to the next without any incompatibility issues. Lastly, in addition to having a sustainable work environment, it\u2019s crucial for AI developers to obtain the right data that has its uses not only during ML training, but also during evaluation and monitoring, both pre- and post-deployment (i.e., before and after an image recognition application hits the market).\"}),/*#__PURE__*/e(\"p\",{children:\"When done right, image recognition can have far-reaching, industry-disrupting effects on different ML-backed fields, among them e-commerce, autonomous driving, smartphone apps, digital media, as well as search-and-rescue operations involving surveillance drones. We\u2019re thrilled that these cutting-edge image recognition and object detection technologies are being designed, built, and implemented by many of our requesters the world over, with the data- and platform-related support coming from Toloka\u2019s own crowd contributors, data scientists, and ML engineers.\"})]});export const richText4=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"A review of models based on real-world applications\"})}),/*#__PURE__*/e(\"p\",{children:\"Modern large language models (LLMs) have demonstrated the best performance for many NLP tasks from text classification to text generation. But are they really a \u201Csilver bullet\u201D or a one-stop-shop solution? Can they be applied across the board? Toloka's ML team faces these kinds of tasks all the time, and our answer so far is a resounding \u201CNo.\u201D Performance is not the only factor you should be concerned about when developing a model for a real use case. And you probably don\u2019t want to spend your entire department\u2019s budget on it either. \"}),/*#__PURE__*/e(\"p\",{children:\"We've created a practical guide on ways to solve text classification problems \u2013 depending on how much data you have, the type of data (short or long texts, common or specific topics, etc.), time and budget constraints, computational and security requirements, and other factors.\"}),/*#__PURE__*/e(\"h2\",{children:\"Approaches to text classification\"}),/*#__PURE__*/e(\"p\",{children:\"Let\u2019s start off with a brief overview of potential models and solutions you could use.\"}),/*#__PURE__*/e(\"h3\",{children:\"Old-school tf-idf models\"}),/*#__PURE__*/e(\"p\",{children:\"Models in this category are founded on basic stats like word count and co-occurrences. Reduced feature space is usually passed to one of the classic ML models like SVM, MLP or Naive Bayes. This method is easy to implement and does not require any specific libraries or accelerators \u2014 you\u2019re good to go with one of the classic solutions like sklearn or NLTK. Moreover, these models can easily handle both short and long texts. Given their relatively small size, they\u2019re highly efficient when it comes to training, deployment, and inference.\"}),/*#__PURE__*/e(\"p\",{children:\"Nevertheless, this approach has several drawbacks, the most important one being performance. Compared to other approaches, tf-idf models rank the lowest. Additionally, you\u2019ll have to carry out extensive preprocessing of your texts (misspellings, stopwords, punctuation, lemmatization, and more). You also need a lot of data to produce a robust model \u2014 and your data should resemble academic texts, not social media, which can contain numerous misspellings as well as slang.\"}),/*#__PURE__*/e(\"h3\",{children:\"First embeddings and pre-trains\"}),/*#__PURE__*/e(\"p\",{children:\"Word embeddings are excellent for text classification since each word (or sequence of characters) is represented by a vector of numbers containing useful information about the context, use, and semantics.\"}),/*#__PURE__*/e(\"p\",{children:\"Take for example Word2Vec, its variations and implementation within the fastText library \u2014 a binary file that you can run to achieve a desired result. While you still need a large dataset to produce solid word embeddings and to train the classifier head, with proper configuration you can significantly reduce your preprocessing efforts. As a single multiprocessing library, it can run right in the system\u2019s console.\"}),/*#__PURE__*/e(\"p\",{children:\"Training time increases with model size (as you have to learn and save an embedding for every token and classification head\u2019s weights), but it is still at an acceptable level. On average, it takes anywhere from 10 minutes to an hour to create a fastText model between 300 MB and 2 GB. This model can handle texts of any size, and the inference is incredibly fast given that the sole focus is on text embedding construction and processing by an MLP. The availability of pre-trained word embeddings for a variety of languages makes it a baseline for almost any text classification problem.\"}),/*#__PURE__*/e(\"h3\",{children:\"Small transformers\"}),/*#__PURE__*/e(\"p\",{children:\"This category includes transformer-based language models such as BERT and RoBERTa \u2014 currently considered to be state-of-the-art NLP. Even though it may seem obvious that a model with 110 million parameters is \u201Csmall\u201D, and a model with 175 billion parameters is \u201Clarge\u201D, it\u2019s not easy to distinguish between small and large transformers. Yet there are several key advantages that make transformers a great option. Namely, they\u2019re resistant to misspellings and usually require little preprocessing compared to other models.\"}),/*#__PURE__*/e(\"p\",{children:\"Since you probably won\u2019t be training your own BERT and will likely be using a pre-trained model from a library or hub (like Hugging Face), you can use comparatively small datasets to create decent models. If you have a common task, and your domain is similar to one that already has a tuned version, you may only need a couple hundred or thousand samples to slightly tune the model and achieve great results. The model size usually ranges from 600 MB to several gigabytes. It\u2019s also a good idea to have access to GPUs because the training may take some time to complete.\"}),/*#__PURE__*/e(\"p\",{children:\"However, there are also some disadvantages to consider. The produced model is much slower compared to Word2Vec, so if you require real-time inference you\u2019ll need to either use a GPU device or invest in model optimization (graph optimizations, ONNX, DeepSpeed, and others). Additionally, the length of texts that a model can handle is limited by its architecture and is usually about 512 tokens (which corresponds to approximately 380 words), and it is up to you to decide from which part of your text these tokens should be taken. In practice, a simple approach like taking 192 tokens from the beginning works well.\"}),/*#__PURE__*/e(\"h3\",{children:\"LLMs\"}),/*#__PURE__*/e(\"p\",{children:\"It\u2019s likely that you don\u2019t have your own LLM \u2014 they\u2019re really big! The size of the large downloadable version of T5 is about 40 GB. You\u2019ll have to deploy this model somehow and inference may take time. In which case, you\u2019ll either need to use an expensive computational cluster or opt for a service that provides an API, like OpenAI with its GPT-3 model.\"}),/*#__PURE__*/e(\"p\",{children:\"One benefit is that LLMs require little data for tuning, and you don\u2019t need to worry about preprocessing. As a side note, approaches like zero-shot or few-shot don't work well for text classification problems. You\u2019ll need to either fine-tune or p-tune (prompt-tune) your model. If you choose to use an API, you\u2019ll also need to consider internet access, data security, SLAs, pricing, and more. However, achievable performance from using LLMs is the biggest plus.\"}),/*#__PURE__*/e(\"h2\",{children:\"Choosing by scenario\"}),/*#__PURE__*/e(\"p\",{children:\"As expected, all these approaches have their own pros and cons. Consider a variety of architectures to find a good fit for your specific real-world task. We recommend basing your decision on the actual requirements you have for your text classification problem.\"}),/*#__PURE__*/e(\"p\",{children:\"Let\u2019s go over some of the most common cases we\u2019ve encountered and our recommended approaches. Your text classification task will likely fall under one of these scenarios.\"}),/*#__PURE__*/e(\"h3\",{children:\"Your goal is to create a high-performing model\"}),/*#__PURE__*/e(\"p\",{children:\"If performance really matters, choose a transformer. Spend some time searching for the optimal architecture and pre-trained weights, expanding your dataset, optimizing your pipeline and parameters, and so on. Also, try tuning an LLM, either your own or via an API. Just know that it will take time. You need to have expertise in ML and/or NLP to achieve the best results in this case.\"}),/*#__PURE__*/e(\"h3\",{children:\"You have little data available\"}),/*#__PURE__*/e(\"p\",{children:\"Go for LLMs or a small tuned transformer. If your task is general enough, you can leverage extensive model catalogs that are available across various hubs.\"}),/*#__PURE__*/e(\"h3\",{children:\"You have a lot of data available\"}),/*#__PURE__*/e(\"p\",{children:\"Start with fastText and establish a baseline. This may be enough if your performance requirements aren\u2019t that strict. If they are, go for the fine-tuning process of one of the small pre-trained transformers. If an API is an option and you have a dedicated budget, you can try tuning an LLM too.\"}),/*#__PURE__*/e(\"h3\",{children:\"You have privacy or security concerns about your data\"}),/*#__PURE__*/e(\"p\",{children:\"If you have privacy concerns, you don\u2019t want your data to leave a specific contour and be logged by a third-party service. An API is not an option until you make your logging and security concerns clear to the provider. Choose local models that you can deploy yourself according to your hardware and software setup. Also keep in mind that data and model locations are important due to modern privacy legislations.\"}),/*#__PURE__*/e(\"h3\",{children:\"You have a common task and domain\"}),/*#__PURE__*/e(\"p\",{children:\"Someone has probably already solved the task for you and you can apply their solutions. Simply look for applicable tuned transformers. LLMs will likely work too, but we\u2019ve noticed that previously tuned transformers can outperform LLMs if the dataset is extremely small (a couple hundred samples). However, the difference is minute.\"}),/*#__PURE__*/e(\"h3\",{children:\"Your task or data is very specific\"}),/*#__PURE__*/e(\"p\",{children:\"In this case, LLMs have the best performance compared to other approaches. Training an adequate small transformer is a challenge under these circumstances, and other architectures usually perform much worse.\"}),/*#__PURE__*/e(\"h3\",{children:\"Your model will be used for online inference (under hundreds of milliseconds)\"}),/*#__PURE__*/e(\"p\",{children:\"Try fastText because of its speed. If you\u2019re not satisfied with the quality, you can try using a small transformer. However, you\u2019ll most likely have to use an optimization mechanism or deploy your model with access to a GPU. There are lots of ways to speed up inference with BERT-like models \u2014 even a brief description would take an entire article. Some of them may be implemented with just 1 parameter change or 3 lines of code. Others require rewriting of the whole model and training pipeline. LLMs are usually not an option here unless they\u2019ve been optimized.\"}),/*#__PURE__*/e(\"h3\",{children:\"Your model will be used for batch processing only\"}),/*#__PURE__*/e(\"p\",{children:\"Opt for a large model (or a small transformer without any optimizations). While it seems straightforward at first glance, you still need an understanding of timing so that your batches don\u2019t stack up.\"}),/*#__PURE__*/e(\"h3\",{children:\"You\u2019re concerned about scalability\"}),/*#__PURE__*/e(\"p\",{children:\"For example, your model will be widely used, and you expect high RPS on its endpoint. You\u2019ll probably apply an orchestration mechanism like Kubernetes, assuming that your pods can be deployed and destroyed quickly, in which case there may be restrictions on model size (namely, image size). Therefore, fastText and small transformers are common options.\"}),/*#__PURE__*/e(\"h3\",{children:\"You have access to a computational cluster with modern GPUs\"}),/*#__PURE__*/e(\"p\",{children:\"If so, you're lucky! You can play with different types of transformers, even large ones, but the real question is, can you use a node pool with GPUs for inference? If the answer is yes, you can choose whatever you want, even LLMs. If not, you\u2019ll probably find yourself optimizing a small transformer.\"}),/*#__PURE__*/e(\"h3\",{children:\"You have no access to modern hardware accelerators\"}),/*#__PURE__*/e(\"p\",{children:\"That\u2019s unfortunate to hear! Start with basic approaches and train a fastText model. You can also train a transformer model in this setup, but it will require a deeper understanding of optimization mechanisms. Another option is to move from classic libraries into something more specific like FasterTransformer.\"}),/*#__PURE__*/e(\"h3\",{children:\"You have a lot of time to build a model\"}),/*#__PURE__*/e(\"p\",{children:\"Try any architecture, different pre-trains or parameters, and perform \u201Closs-watching\u201D.\"}),/*#__PURE__*/e(\"h3\",{children:\"You have almost no time to build a model\"}),/*#__PURE__*/e(\"p\",{children:\"In this case, fastText and API-accessible LLMs are good options. If your task is popular, you can tune an appropriate small transformer with a default set of hyperparameters. Still, API-accessible LLMs are usually the best choice.\"}),/*#__PURE__*/e(\"h3\",{children:\"You need to create N models per day with good performance, and preferably automate the process\"}),/*#__PURE__*/e(\"p\",{children:\"This scenario imposes size and computational limits on your models. Small or optimized models are a good fit. It won\u2019t be reasonable to tune an LLM on a regular basis, or to store and deploy hundreds of copies of it. Another good option could be fastText, if it achieves sufficient performance.\"}),/*#__PURE__*/e(\"h2\",{children:\"A real-life story\"}),/*#__PURE__*/e(\"p\",{children:\"So, what can go wrong if you choose an inappropriate architecture, apart from insufficient performance?\"}),/*#__PURE__*/e(\"p\",{children:\"In one of our cases we had two key requirements for the classification model: good performance and the ability to handle text batches of 100 to 1000 texts every ten minutes. We trained a small transformer model, deployed it as an online-endpoint on a Kubernetes cluster, and everything worked just fine\u2026 Until an extremely large batch arrived.\"}),/*#__PURE__*/e(\"p\",{children:\"It caused a system-wide hang for more than an hour, resulting in timeout exceptions for other batches. We realized that we got the wrong request statistics, and we had to properly configure an auto scaling mechanism on the cluster and invest more time and effort into model optimization. This is just one example of risks you can mitigate by careful planning.\"}),/*#__PURE__*/e(\"h2\",{children:\"What it all comes down to\"}),/*#__PURE__*/e(\"p\",{children:\"In the end, there\u2019s no \u201Csilver bullet\u201D or \u201Cone-size-fits-all\u201D approach. As a key takeaway, try to avoid looking at your text classification challenge with only performance in mind. There are other factors that are worth considering, like inference and training time, budget, scalability, privacy, and data type.\"}),/*#__PURE__*/e(\"h2\",{children:\"Connect with us\"}),/*#__PURE__*/t(\"p\",{children:[\"Feel free to reach out via \",/*#__PURE__*/e(a,{href:\"https://join.slack.com/t/tolokacommunity/shared_invite/zt-1p5zts2rb-7wqVYJllzWD74sVbaxON9w\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Slack\"})}),\" if you\u2019d like to talk about these solutions and more. We\u2019re always happy to hear from you!\"]}),/*#__PURE__*/e(\"h2\",{children:\"About Toloka\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Toloka is a European company based in Amsterdam, the Netherlands that provides data for Generative AI development. Toloka empowers businesses to build high quality, safe, and responsible AI. We are the trusted data partner for all stages of AI development from training to evaluation. Toloka has over a decade of experience supporting clients with its unique methodology and optimal combination of machine learning technology and human expertise, offering the highest quality and scalability in the market.\"})})]});export const richText5=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"A guide to data annotation services If you have heard of data annotation (or labeling) services and want to know more about how they might help you, your company, or your industry, this article is for you. We plan to provide some answers: what these services are (with several examples of use cases), who can benefit from them, and how.\"}),/*#__PURE__*/e(\"h2\",{children:\"What are data annotation services?\"}),/*#__PURE__*/e(\"p\",{children:'\"Annotation\" is the process of adding comments, notes, or explanations to something, for example, images or text. Data annotation is the process of labeling data, be it text, images, videos, audio, or any other format, so that computers can understand and interpret that data.'}),/*#__PURE__*/t(\"p\",{children:[\"Companies provide data annotation services and outsource the process of taking large quantities of data and then annotating it. The annotation process can include data labelling, categorizing unstructured data, correcting, verifying, and more. Check out our \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/labeling-data-for-machine-learning/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"article on data labeling\"})}),\" to learn more.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Who uses data labeling services and why?\"}),/*#__PURE__*/e(\"p\",{children:\"Data annotation is absolutely necessary for anyone developing artificial intelligence and machine learning algorithms, which rely on supervised learning: companies, researchers, ML engineers, and more. Products and tools powered by machine learning are only as good as the data they are trained with. So, when a company using machine learning wants to improve their ML model, they will naturally look to their training data. Whether it's sourcing new data or adding to and improving the quality of existing data, there are many organizations ready to outsource this important, but potentially resource-intensive task to a data annotation company.\"}),/*#__PURE__*/e(\"p\",{children:\"And nowadays, it's not just large, cutting-edge tech companies who require this service. Machine learning is already running in the background of so many elements of modern life.\"}),/*#__PURE__*/e(\"p\",{children:\"Data annotation is especially popular with certain industries, such as:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Retail and e-commerce.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Social science research.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Product & marketing research.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Healthcare and automation of medical data processing.\"})})]}),/*#__PURE__*/e(\"p\",{children:'Mainstream sectors like retail and marketing are using data annotation to attract new and repeat customers. Retail and marketing sectors may outsource the acquisition of crucial data to a data annotation company and use it to improve their online search relevance or recommender system (\"people who bought this product, also liked these products\").'}),/*#__PURE__*/e(\"h2\",{children:\"How do data annotation services work?\"}),/*#__PURE__*/e(\"p\",{children:\"There are three main approaches to fulfilling data labeling needs: companies can do it with their in-house team (their own employees label the data), outsource it (a company that provides data collection and/or data annotation), or use a data annotation platform.\"}),/*#__PURE__*/e(\"p\",{children:\"The latter approach is known as crowdsourcing, and for it to work, you need three key components:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Requesters\"}),\" \u2013 those who need to collect high-quality data. To get high-quality labels, the requester needs to create labeling tasks that are easy to understand and complete, which involves breaking them down into more manageable parts and writing clear instructions. This can be a difficult job to do, but that's what platforms like Toloka are for \u2013 our data annotation experts can help requesters build data annotation pipelines that work.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Annotators\"}),\" \u2013 large numbers of individuals, from across the globe, who will annotate and label data for a small reward. Since the tasks are decomposed into smaller parts, even people without any particular qualification or skill set can complete them successfully. That being said, many data annotation tasks filter who can complete them, so that only annotators with a certain education, language, location, device, age, or gender can complete that specific task.\"]}),/*#__PURE__*/t(\"p\",{children:[\"A \",/*#__PURE__*/e(\"strong\",{children:\"platform\"}),\" like \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/data-labeling-platform/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Toloka\"})}),\", which brings requesters and annotators together.\"]}),/*#__PURE__*/e(\"h2\",{children:\"What is Toloka?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://toloka.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Toloka\"})}),\" provides a unified environment to support fast and scalable AI/ML development. Among many other things, Toloka uses crowdsourcing and microtasking to provide data labeling and annotation services, at scale.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Toloka's \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/data-labeling-platform/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"data labeling platform\"})}),\" can be used by businesses and researchers to collect or label data to optimize machine learning models (used in anything from voice recognition to autonomous vehicles and search algorithms). The quality and size of a dataset is crucial to maximize the accuracy of a machine learning model.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Data labeling tasks\"}),/*#__PURE__*/e(\"p\",{children:\"We can break down the kinds of data annotation services available on Toloka's platform into the following data labeling subgroups:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Image annotation\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Video annotation\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Audio annotation\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Texts annotation\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Data enrichment\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Surveys\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Field tasks\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Before annotators, or Tolokers as we call them, can begin their work, the requester needs to create a task for them: set up the task interface (what the task will look like), and provide instructions and examples for the Tolokers to follow. The requester can then specify several attributes they might desire of their Tolokers \u2013 such as location or language, for instance, to avoid biased results and improve data diversity. The requester can also then add quality control methods, such as golden sets or post-verification, to ensure the data they receive in the end is the standard they require.\"}),/*#__PURE__*/e(\"p\",{children:\"Now we will discuss in more detail what sort of tasks Tolokers are asked to complete, and what those tasks look like.\"}),/*#__PURE__*/e(\"h2\",{children:\"I am not a robot\"}),/*#__PURE__*/e(\"p\",{children:\"These first couple of example tasks we're picking out are ones you may already be familiar with:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Recognize text in an image.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Recognize objects in a photo.\"})})]}),/*#__PURE__*/e(\"p\",{children:'The data here are images. Object detection in an image can look like a multiple-choice array of images, where the Toloker needs to select the images with a certain object in them. Many websites ask you to complete similar tasks to \"prove you\\'re not a robot\". Such labeled data is frequently used in computer vision models.'}),/*#__PURE__*/e(\"p\",{children:\"Another kind of object recognition task is one where Tolokers see a single image at a time and need to identify and outline specific objects within that image (there is an editor for them to draw an outline). Requesters can then take all the annotated images and use this dataset for training computer vision software.\"}),/*#__PURE__*/e(\"h2\",{children:\"Data labeling task examples\"}),/*#__PURE__*/e(\"p\",{children:\"Below we cover some more common tasks for the main data labeling case subgroups, and with most tasks it's easy to understand why that information would be useful to these industries:\"}),/*#__PURE__*/e(\"h3\",{children:\"Image annotation\"}),/*#__PURE__*/e(\"p\",{children:'A \"Product search relevance\" task presents Tolokers with an image, a box for the search query, and radio buttons \u2013 for example, \"relevant\" or \"not relevant\". This type of data annotation will help to improve ML algorithms behind e-commerce search page results by rating how relevant products are to specific search queries.'}),/*#__PURE__*/e(\"p\",{children:'In an \"Image comparison (Side-by-side)\", Tolokers are asked to compare two side-by-side images and select one of the possible response options. Requesters might use this type of image annotation service when they need to collect opinions from large groups of people, for example to:'}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Improve user experience by understanding which design users like best.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Test out which images are more impactful for targeted ads.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Decide on the best images for various kinds of publications.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Other popular tasks include image classification, selecting objects with bounding boxes, polygons, or key points, and so on.\"}),/*#__PURE__*/e(\"h3\",{children:\"Video annotation\"}),/*#__PURE__*/e(\"p\",{children:'\"Video classification\" is the type of data annotation task where Tolokers are given a video player and a few different response options. Tolokers must watch the video clip and select from the available response options. Requesters use this kind of task to get training data for several purposes:'}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Sorting video clips into categories.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Moderating content.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Detecting imperfections in the video \u2013 both audio and visual.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Rating video clips based on levels of enjoyment or other measures.\"})})]}),/*#__PURE__*/e(\"p\",{children:'A \"Side-by-side\" comparison of two videos could be used to help editors or marketing specialists decide which is the best opening frame for a video, or to help decide which video looks more realistic. Of course, it can also be used to train ML models to make decisions like that on their own.'}),/*#__PURE__*/e(\"h3\",{children:\"Text annotation\"}),/*#__PURE__*/e(\"p\",{children:'A \"Generating product descriptions\" task can help e-commerce companies to quickly write many accurate product descriptions. For this task, the Toloker is presented with an image of the product and the product name and asked to come up with a product description to put in the text input area.'}),/*#__PURE__*/e(\"p\",{children:'Other text-based tasks focus on \"Sentiment analysis and content moderation\". These tasks provide Tolokers with a text and several response options. Tolokers read the text and select from the available options (they might then add specifics via an additional question with more checkboxes). The requesters\\' aims for this task can be varied. They might want their ML model to learn to:'}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"To moderate comments and nicknames on social media.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"To check product reviews in a store, ads on a site, or posts on social networks.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"To check for the presence of a brand or company name.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"There are various end uses for the datasets created: improving user/customer experience; complying with regulations; conducting brand and consumer research \u2013 to name a few. There are other popular types of text annotation used for natural language processing, for example text classification, semantic segmentation, named entity recognition, and more.\"}),/*#__PURE__*/e(\"h3\",{children:\"Audio annotation\"}),/*#__PURE__*/e(\"p\",{children:'\"Transcribing audio recordings\" tasks provide the data needed to improve speech recognition models. Tolokers completing these tasks see only an audio player and a text input area: they listen to the recording and write down what they hear. The same Tolokers might transcribe many different recordings, or many Tolokers might transcribe the same recording. Either way, the mass of transcriptions produced is vital annotated data from which machines can \"learn\" to accurately interpret speech \u2013 meaning less mistakes made by voice recognition software and thereby improving communications in numerous fields.'}),/*#__PURE__*/e(\"p\",{children:'This is an example of the kind of task where a requester should specify a certain attribute for the Tolokers, like \"language\" (education, skillset, and device may also be relevant depending on the content and format of the audio recording).'}),/*#__PURE__*/e(\"p\",{children:\"Other kinds of audio tasks include:\"}),/*#__PURE__*/e(\"p\",{children:'\"Audio classification\" \u2013 where Tolokers are again given an audio player but, in place of a free-type input area, there is a choice of given answers, from which they must choose.'}),/*#__PURE__*/e(\"p\",{children:'\"Voice recording\" \u2013 where Tolokers are given text to read aloud and a voice recorder button. Datasets produced by the latter can be useful in training text to speech software such as that used in translation apps.'}),/*#__PURE__*/e(\"h3\",{children:\"Field tasks\"}),/*#__PURE__*/e(\"p\",{children:'Field tasks send our most intrepid Tolokers out on a mini mission. For example, a \"Spatial Crowdsourcing\" task (completed in the Toloka mobile app) requires Tolokers to select a point on the map, go to the location, take photos and/or write a comment.'}),/*#__PURE__*/e(\"p\",{children:\"The aim of a Field Task might not be connected to Machine Learning, but instead to improving processes and quality assurance. For example, an urban field task might ask Tolokers to go to the entrance of metro stations to take a photo and assess the cleanliness of the metro station. Several potential requesters would find this data useful \u2013 digital maps, transport companies, city councils, and city tourism boards.\"}),/*#__PURE__*/e(\"h3\",{children:\"Data enrichment\"}),/*#__PURE__*/e(\"p\",{children:'Another popular task among e-commerce requesters is a \"Product photo search\". Tolokers might be asked to search for product photos online or to search for instances of a fashion brand\\'s logo. This task gives Tolokers links to search online, a product description, and an upload area.'}),/*#__PURE__*/e(\"p\",{children:\"The dataset produced can be used in brand awareness research which is crucial to influencing the marketing team activities behind any big brand.\"}),/*#__PURE__*/e(\"h2\",{children:\"The benefits of high-quality training data\"}),/*#__PURE__*/e(\"p\",{children:\"Most of the requesters using Toloka are working with machine learning models, which either enhance their business and product or may form the whole basis for their product. Today's world has countless uses for machine learning \u2013 from image and speech recognition to weather modeling and medical diagnoses, and even self-driving cars.\"}),/*#__PURE__*/e(\"p\",{children:\"Machine learning models are only as good as the data they learn from. The more data you have, and the better quality that data is, the more accurate your ML model will be. Better data leads to better machine learning, which leads to better real-world practical and financial results.\"}),/*#__PURE__*/e(\"p\",{children:\"Starting off with high quality human-labeled training data ultimately means search algorithms are quicker and more accurate; it means that speech recognition software makes fewer mistakes; and your self-driving car is safer.\"}),/*#__PURE__*/e(\"p\",{children:\"Getting humans to process and label large quantities of data could be incredibly expensive and time-consuming. Businesses, data scientists, and other individuals could end up making sacrifices in quality and accuracy, simply to avoid such costs.\"}),/*#__PURE__*/e(\"p\",{children:\"Now, the possibility of outsourcing data annotation services to an efficient data annotation company is key to the viability of accessing human-labeled data in high quantities.\"}),/*#__PURE__*/e(\"h2\",{children:\"Can data annotation services help you?\"}),/*#__PURE__*/e(\"p\",{children:\"We hope that by giving you some information on the type of data annotation services offered by Toloka, you now have a clearer understanding about what a data labeling service is and that you have some ideas about the potential uses for you and your industry.\"})]});export const richText6=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"h2\",{children:\"About Taboola\"}),/*#__PURE__*/e(\"p\",{children:\"Taboola is one of the world's largest content recommendation platforms, curating online content for users based on their personal interests. Their two main types of stakeholders are content providers such as CNN, NBC, and MSN.com and advertisers who pay to have their ads displayed in the newsfeed. \"}),/*#__PURE__*/e(\"img\",{alt:\"About Taboola\",className:\"framer-image\",height:\"706\",src:\"https://framerusercontent.com/images/ETqxOVq8YoNcwz5T5Qiz2GTowM.webp\",srcSet:\"https://framerusercontent.com/images/ETqxOVq8YoNcwz5T5Qiz2GTowM.webp?scale-down-to=512 512w,https://framerusercontent.com/images/ETqxOVq8YoNcwz5T5Qiz2GTowM.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/ETqxOVq8YoNcwz5T5Qiz2GTowM.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/ETqxOVq8YoNcwz5T5Qiz2GTowM.webp 2560w\",style:{aspectRatio:\"2560 / 1413\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:\"One of Taboola\u2019s priorities is to regulate the ad content on their website. They have an in-house team of content moderators dedicated to this task, but they face several challenges. With Toloka, they were able to scale an AI moderation solution and reduce the need for manual review.\"}),/*#__PURE__*/e(\"h2\",{children:\"Challenge: Policy rules and content moderation\"}),/*#__PURE__*/e(\"p\",{children:\"Major publishers do not want their newsfeeds to include disturbing, fake, or adult content. Therefore, as a service provider, Taboola\u2019s first goal is to prevent the display of these images on the network.\"}),/*#__PURE__*/e(\"p\",{children:\"The second goal is to label ads by category. Because Taboola partners with advertisers across a wide variety of ad categories and quality tiers, news publishers can choose which types of ads will appear in their feeds. For instance, companies whose content is targeted at children, like Disney, may want to exclude ads for alcoholic beverages. It\u2019s important for the content moderation team to label ads correctly so that they don't end up in the wrong feed.\"}),/*#__PURE__*/e(\"img\",{alt:\"Challenge: Policy rules and content moderation\",className:\"framer-image\",height:\"706\",src:\"https://framerusercontent.com/images/Jr1FiA9Xsqey0ZMjX2q8u1oAoc.webp\",srcSet:\"https://framerusercontent.com/images/Jr1FiA9Xsqey0ZMjX2q8u1oAoc.webp?scale-down-to=512 512w,https://framerusercontent.com/images/Jr1FiA9Xsqey0ZMjX2q8u1oAoc.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/Jr1FiA9Xsqey0ZMjX2q8u1oAoc.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/Jr1FiA9Xsqey0ZMjX2q8u1oAoc.webp 2560w\",style:{aspectRatio:\"2560 / 1413\"},width:\"1280\"}),/*#__PURE__*/e(\"h2\",{children:\"Taboola\u2019s moderation pipeline\"}),/*#__PURE__*/e(\"p\",{children:\"To overcome its main moderation challenge \u2014 an in-house team receiving too many images for manual review \u2014 and to make the process easily scalable, Taboola employs a combination of technologies, including automation, AI, and human labeling. Let\u2019s take a closer look at each of them.\"}),/*#__PURE__*/e(\"p\",{children:\"Automation works well when there are predefined rules, and the decisions are relatively straightforward. For example, if an image that has been rejected by human moderation shows up again, automation will easily identify it as non-compliant and reject it.\"}),/*#__PURE__*/e(\"p\",{children:\"The second stage uses ML models, which have previously proven effective in making decisions on standard issues like image safety. How does it work? When an image is uploaded to the model, it first detects and highlights problematic areas, such as adult content, and then decides whether the image is compliant. If it isn't, it's \\\"red flagged.\\\"\"}),/*#__PURE__*/e(\"p\",{children:\"On top of these technologies is the human evaluation performed by the Content Review team, who deal with the most difficult, vague, and risky cases. To optimize the manual review process, the Taboola team identified an area where AI could help reduce the workload: identifying similar images.\"}),/*#__PURE__*/e(\"p\",{children:\"In Taboola\u2019s experience, rejected images tend to come back to the platform repeatedly. Some advertisers try to trick the system by submitting slightly modified versions of the same image over and over again, which means the image is returned for manual review multiple times. To address this issue and improve the efficiency of ad moderation, Taboola created an AI-powered image similarity model that can recognize repeat images and reject them without manual review.\"}),/*#__PURE__*/e(\"img\",{alt:\"Taboola\u2019s moderation pipeline\",className:\"framer-image\",height:\"706\",src:\"https://framerusercontent.com/images/L1B6HMXdZvJBagMuX0j89pJJn8.webp\",srcSet:\"https://framerusercontent.com/images/L1B6HMXdZvJBagMuX0j89pJJn8.webp?scale-down-to=512 512w,https://framerusercontent.com/images/L1B6HMXdZvJBagMuX0j89pJJn8.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/L1B6HMXdZvJBagMuX0j89pJJn8.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/L1B6HMXdZvJBagMuX0j89pJJn8.webp 2560w\",style:{aspectRatio:\"2560 / 1413\"},width:\"1280\"}),/*#__PURE__*/e(\"h2\",{children:\"Solution: Automation powered by AI\"}),/*#__PURE__*/e(\"p\",{children:'Basically, the AI-powered automation model was designed to find similarities in images, labeling them as \"similar\" or \u201Cnot similar.\"'}),/*#__PURE__*/e(\"p\",{children:'The operating principle of the model is based on calculating the distance, or length of a line segment, between the two image elements being compared. Images are considered \"similar\" by the model if their distance is less than the set threshold. If the distance is greater than the threshold, the model will label them as \"not similar.\"'}),/*#__PURE__*/e(\"p\",{children:\"However, certain images, such as resized images or images of two people who appear to be identical, were still difficult for the model to identify. Before moving on to the next steps, the team needed to establish ground truth to define the concept of similarity and use it to establish the threshold value for the model.\"}),/*#__PURE__*/e(\"img\",{alt:\"Solution: Automation powered by AI\",className:\"framer-image\",height:\"706\",src:\"https://framerusercontent.com/images/9UDp7HknDP5vcBWxgLAj2eM8TO8.webp\",srcSet:\"https://framerusercontent.com/images/9UDp7HknDP5vcBWxgLAj2eM8TO8.webp?scale-down-to=512 512w,https://framerusercontent.com/images/9UDp7HknDP5vcBWxgLAj2eM8TO8.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/9UDp7HknDP5vcBWxgLAj2eM8TO8.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/9UDp7HknDP5vcBWxgLAj2eM8TO8.webp 2560w\",style:{aspectRatio:\"2560 / 1413\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:\"Toloka\u2019s huge crowd provided the large volume of human evaluations needed for training the model. Each pair of images was given to several trained content reviewers to evaluate, and the majority vote (consensus) was used to determine the ground truth. The next step was to experiment with different threshold values and test the model on a few images to get the best results.\"}),/*#__PURE__*/e(\"h2\",{children:\"Results\"}),/*#__PURE__*/e(\"p\",{children:\"The image similarity model proved to be effective and the results of the work quickly paid off. With this model in place, AI-powered automation processes now cover 11% of all ads, compared to 4% before. Manual review was reduced from 31% to 20%, freeing up time for in-house content moderators to focus on more important tasks such as providing better customer service. \"}),/*#__PURE__*/e(\"img\",{alt:\"Results\",className:\"framer-image\",height:\"706\",src:\"https://framerusercontent.com/images/0pdktXBqN68pRPEemd3NvwTVq0.webp\",srcSet:\"https://framerusercontent.com/images/0pdktXBqN68pRPEemd3NvwTVq0.webp?scale-down-to=512 512w,https://framerusercontent.com/images/0pdktXBqN68pRPEemd3NvwTVq0.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/0pdktXBqN68pRPEemd3NvwTVq0.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/0pdktXBqN68pRPEemd3NvwTVq0.webp 2560w\",style:{aspectRatio:\"2560 / 1413\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[\"The case was presented at the \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/events/content-moderation\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Data-Driven AI meetup\"})}),\" by Gal Cohen, Product Manager, Taboola. The full video is available \",/*#__PURE__*/e(a,{href:\"https://www.youtube.com/watch?v=OCiLsqt3Tqg\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"here\"})}),\".\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(a,{href:\"https://tlkfrontprod.azureedge.net/portal-production/static/files/content-moderation-meetup.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Download presentation\"})})})]});\nexport const __FramerMetadata__ = {\"exports\":{\"richText1\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText2\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText6\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText4\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText5\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText3\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"__FramerMetadata__\":{\"type\":\"variable\"}}}"],
  "mappings": "+LAAsJ,IAAMA,EAAsBC,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,8RAA8R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qdAAqd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sUAAsU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oOAAoO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uSAAuS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4NAA4N,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oYAAoY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sTAAsT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0WAA0W,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wVAAwV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mOAAmO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8bAA8b,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gBAAgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,odAAod,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uWAAuW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2CAA2C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8OAA8O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4IAA4I,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sbAAsb,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wTAAwT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0XAA0X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yUAAyU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qXAAqX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2SAA2S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0aAA0a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gUAAgU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iUAAiU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0QAA0Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iJAAiJ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8ZAA8Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mXAAmX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wBAAwB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yIAAyI,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mFAAmF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+EAA+E,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kDAAkD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qOAAqO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iMAAiM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iEAAiE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kGAAkG,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kDAAkD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8LAA8L,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mNAAmN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wDAAwD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kCAAkC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6KAA6K,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uRAAuR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kOAAkO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gcAAgc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0aAA0a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oaAAoa,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yQAAyQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sOAAsO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uTAAuT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iSAAiS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAoX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gBAAgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uRAAuR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iJAAiJ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAAsP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6hBAA6hB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6RAA6R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,seAAse,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uTAAuT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAAsP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4OAA4O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yZAAyZ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8bAA8b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0VAA0V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sQAAsQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iVAAiV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAA6T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6QAA6Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAoX,CAAC,CAAC,CAAC,CAAC,EAAeC,EAAuBH,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,4WAAuW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6eAA6e,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ipBAAipB,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,6BAA6B,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iQAAiQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wVAAwV,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,yEAAyE,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,wEAAwE,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAqW,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,0EAAgE,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,yEAA+D,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ykBAAokB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,2UAAwVE,EAAEE,EAAE,CAAC,KAAK,qDAAqD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAE,yRAAyR,CAAC,CAAC,EAAeH,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,eAA4BE,EAAEE,EAAE,CAAC,KAAK,6CAA6C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,6nBAA6nB,CAAC,CAAC,EAAeH,EAAE,IAAI,CAAC,SAAS,sRAAsR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,oYAAiZE,EAAEE,EAAE,CAAC,KAAK,2GAA2G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,oCAAoC,CAAC,CAAC,EAAeH,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mfAAqf,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,wDAAwD,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,uDAAuD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qqBAAgqB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uHAAuH,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,yEAAyE,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,wEAAwE,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mIAAmI,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4SAAuS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0JAA0J,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8EAA8E,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,+DAA+D,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,8DAA8D,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,yDAAyD,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,wDAAwD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wdAAwd,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2gBAAsgB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mBAAc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+cAAqc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4WAA4W,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,oGAAoG,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sJAAsJ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,khBAAwgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ibAA4a,CAAC,CAAC,CAAC,CAAC,EAAeI,EAAuBN,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,wKAAwK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ybAAob,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6OAA6O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8XAA8X,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0DAA0D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gQAAgQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8LAA8L,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4XAA4X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6NAA6N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mTAAmT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oFAAoF,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,UAAU,CAAC,EAAE,sIAAsI,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,UAAU,CAAC,EAAE,2NAA2N,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4DAAyEE,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,EAAE,kKAAkK,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,snBAAsnB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iTAAiT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6CAA6C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kQAAkQ,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2GAA2G,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oFAAoF,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2EAA2E,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qIAAqI,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kXAAkX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uSAAuS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+aAA+a,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yeAAye,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mTAA8S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+VAA+V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uJAAuJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uCAAuC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qSAAqS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oWAAoW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8ZAA8Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8OAA8O,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wMAAwM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mSAAmS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wXAAwX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,8WAA8W,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,sDAAiD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qDAAqD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iPAAiP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gKAAgK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2OAA2O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0PAA0P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4IAA4I,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yOAAyO,CAAC,CAAC,CAAC,CAAC,EAAeK,EAAuBP,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,iUAA4T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ynBAAqmB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sKAAuJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,cAAc,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6DAA6D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8CAA8C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sCAAsC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2BAA2B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4DAA4D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8DAA8D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kCAAkC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yCAAyC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,OAAO,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+BAA+B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kDAAkD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6aAA8Z,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yDAAyD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qCAAqC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yCAAyC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8PAA8P,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mkBAA8jB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6aAAoZ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4DAA4D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uOAAkO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+WAAqW,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,yDAAyD,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,2NAA2N,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4NAA4N,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,uCAAuC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,0CAA0C,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0LAAgL,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,oEAAoE,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,yFAAyF,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2TAA4S,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,yDAAyD,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,qDAAqD,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yPAA0O,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,iCAAiC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,0DAA0D,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0JAA0J,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,kFAAkF,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,qIAAqI,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,cAAc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2WAAmU,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,uCAAuC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,0EAA0E,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8FAA8F,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wGAAwG,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,cAAc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yNAAyN,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,8FAA8F,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,wEAAwE,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wNAA8M,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,yHAAyH,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,6EAA6E,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kUAA6T,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,kEAAkE,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,8GAA8G,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sRAAsR,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,wGAAwG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,oGAAoG,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kcAAwb,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oZAA+Y,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,maAAma,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iVAA4U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mdAAmd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6OAA6O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mZAA8Y,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yOAAoO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gdAAsc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8NAA8N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mPAAoO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qdAAsc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yXAAyX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yNAAyN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iYAA4X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mhBAAygB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ghBAAghB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2TAAsT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2TAAsT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4dAAud,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mOAA8N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sZAA4Y,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mlBAA8kB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qSAAgS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4QAA6P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8VAA8V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mjBAA8iB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ycAA+b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6aAA6a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uXAAuX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0QAA0Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+VAA+V,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+ZAA0Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+lBAAglB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8jBAAojB,CAAC,CAAC,CAAC,CAAC,EAAeM,EAAuBR,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,qDAAqD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2jBAA6hB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6RAAwR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6FAAwF,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4iBAA6hB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qeAA2d,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8MAA8M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4aAAka,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,klBAA6kB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8iBAA2gB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,skBAA4jB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8mBAAymB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kYAAoW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8dAA+c,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uQAAuQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sLAA4K,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kYAAkY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6JAA6J,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6SAAwS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uDAAuD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oaAA+Z,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kVAA6U,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iNAAiN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+EAA+E,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ykBAAqjB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+MAA0M,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wWAAmW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6DAA6D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mTAA8S,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oDAAoD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAAwT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kGAAwF,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wOAAwO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gGAAgG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6SAAwS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yGAAyG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8VAAyV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yWAAyW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kVAAyT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,8BAA2CE,EAAEE,EAAE,CAAC,KAAK,6FAA6F,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,uGAA6F,CAAC,CAAC,EAAeH,EAAE,KAAK,CAAC,SAAS,cAAc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,4fAA4f,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeO,EAAuBT,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,kVAAkV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sRAAsR,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qQAAkRE,EAAEE,EAAE,CAAC,KAAK,6DAA6D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAE,iBAAiB,CAAC,CAAC,EAAeH,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,woBAAwoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oLAAoL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yEAAyE,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+BAA+B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uDAAuD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8VAA8V,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uCAAuC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yQAAyQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mGAAmG,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,EAAE,ybAA+a,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,EAAE,4cAAuc,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,KAAkBE,EAAE,SAAS,CAAC,SAAS,UAAU,CAAC,EAAE,SAAsBA,EAAEE,EAAE,CAAC,KAAK,4CAA4C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,oDAAoD,CAAC,CAAC,EAAeH,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEE,EAAE,CAAC,KAAK,qBAAqB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,iNAAiN,CAAC,CAAC,EAAeL,EAAE,IAAI,CAAC,SAAS,CAAC,YAAyBE,EAAEE,EAAE,CAAC,KAAK,4CAA4C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAE,oSAAoS,CAAC,CAAC,EAAeH,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oIAAoI,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2lBAAslB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uHAAuH,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kGAAkG,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+BAA+B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oUAAqU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gUAAgU,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wLAAwL,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0UAAqU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4RAA4R,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wEAAwE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4DAA4D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8DAA8D,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8HAA8H,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ySAAyS,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sCAAsC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oEAA+D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oEAAoE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sSAAsS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sSAAsS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iYAAkY,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qDAAqD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kFAAkF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uDAAuD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sWAAiW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qmBAAgmB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kPAAkP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wLAAmL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4NAAuN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6PAA6P,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uaAAka,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6RAA8R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kJAAkJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oVAA+U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6RAA6R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kOAAkO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uPAAuP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kLAAkL,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oQAAoQ,CAAC,CAAC,CAAC,CAAC,EAAeQ,EAAuBV,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6SAA6S,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,gBAAgB,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mSAA8R,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mNAA8M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,idAA4c,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,iDAAiD,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2SAA4R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iQAAiQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yVAA2V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sSAAsS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0dAAqd,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,qCAAgC,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2IAAsI,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kVAAkV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kUAAkU,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,qCAAqC,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8XAAyX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAoX,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,UAAU,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iCAA8CE,EAAEE,EAAE,CAAC,KAAK,8CAA8C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAE,wEAAqFH,EAAEE,EAAE,CAAC,KAAK,8CAA8C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,MAAM,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeH,EAAE,IAAI,CAAC,SAAsBA,EAAEE,EAAE,CAAC,KAAK,kGAAkG,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBF,EAAEG,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EACxojIM,EAAqB,CAAC,QAAU,CAAC,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,SAAW,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,mBAAqB,CAAC,KAAO,UAAU,CAAC,CAAC",
  "names": ["richText", "u", "x", "p", "richText1", "Link", "motion", "richText2", "richText3", "richText4", "richText5", "richText6", "__FramerMetadata__"]
}