Small experiments insights from ML and software development. Zola 2025-06-04T00:00:00+00:00 https://nodata.dev/atom.xml 2025-06-04T00:00:00+00:00 2025-06-04T00:00:00+00:00 https://nodata.dev/compilers/distributed-systems/gpu/2025/06/04/beyond-distributed-compilation/ <blockquote> <p><strong>Disclaimer</strong>: This article was generated by artificial intelligence.</p> </blockquote> <h1 id="beyond-distributed-compilation-a-new-paradigm-for-heterogeneous-computing">Beyond Distributed Compilation: A New Paradigm for Heterogeneous Computing<a class="zola-anchor" href="#beyond-distributed-compilation-a-new-paradigm-for-heterogeneous-computing" aria-label="Anchor link for: beyond-distributed-compilation-a-new-paradigm-for-heterogeneous-computing"><i class="fas fa-link"></i></a> </h1> <p><img src="/assets/images/c_language.jpg" alt="The Birth of C Language" /> <em>Figure 1: The complexity of modern distributed systems</em></p> <h2 id="the-challenge">The Challenge<a class="zola-anchor" href="#the-challenge" aria-label="Anchor link for: the-challenge"><i class="fas fa-link"></i></a> </h2> <p>Consider the case of Large Language Models (LLMs). Today, we have specialized systems like vLLM or Text Generation Inference (TGI) that are meticulously optimized for NVIDIA GPUs. These optimizations are carefully tuned for the current GPU architecture, where memory bandwidth is often the limiting factor. But what happens when a new GPU architecture emerges with vastly different characteristics - perhaps one with much higher memory bandwidth and lower latency? Suddenly, all that carefully crafted code needs to be rewritten.</p> <p>The problem extends beyond just GPUs. What if you want to run the same LLM on CPUs, which have entirely different performance characteristics? Or across multiple nodes, where network latency and bandwidth become critical factors? With a high-bandwidth, low-latency InfiniBand connection, you might want one data layout, but with a slower network, you'd need a completely different approach.</p> <p><img src="/assets/llm-bottlenecks.png" alt="LLM Inference vs Training Bottlenecks" /> <em>Figure 2: Different hardware architectures require different optimization strategies</em></p> <p>This situation is strikingly similar to the early days of computing, when each CPU architecture required its own specialized assembly code. Then came C, and with it, a revolutionary idea: write your code once in a system-agnostic way, and let the compiler handle the translation to the specific instruction set. The compiler could optimize for the target architecture, taking into account its unique characteristics.</p> <p>We need a similar revolution for distributed systems. Instead of writing specialized code for each hardware configuration, we need a language that describes the problem at a high level, independent of the underlying hardware. The compiler would then handle the complex task of optimizing for the specific architecture, taking into account:</p> <ul> <li>Memory bandwidth and latency</li> <li>Network topology and characteristics</li> <li>Cache hierarchies</li> <li>Hardware-specific schedulers</li> <li>Data movement patterns</li> </ul> <p>When new hardware emerges, we wouldn't need to rewrite our code - we'd just need to recompile it. The compiler would understand the new architecture's characteristics and generate optimized code accordingly. This would free developers to focus on solving problems rather than wrestling with hardware-specific optimizations.</p> <h2 id="the-problem">The Problem<a class="zola-anchor" href="#the-problem" aria-label="Anchor link for: the-problem"><i class="fas fa-link"></i></a> </h2> <p>The fundamental challenge stems from a theoretical limitation: Turing-complete programming languages face the halting problem. This means it's impossible for a compiler to determine if a program will halt, let alone predict how long it will run. This uncertainty makes it extremely difficult for compilers to reason about program optimization. As a result, developers must manually profile their code, identify hot loops, optimize cache usage, and adjust data structures to fit cache lines - all while understanding the intricate details of their target hardware.</p> <p><img src="/assets/images/turing_vs_pure.jpg" alt="Turing vs Restricted Model" /> <em>Figure 3: The trade-off between expressiveness and optimization potential</em></p> <p>This situation suggests a radical solution: what if we gave up Turing completeness? By adopting a more restricted programming model, we could enable compilers to precisely determine when and how a program will execute. This would allow us to explore all possible execution strategies during compilation and find the optimal one. While finding the absolute optimal solution is likely NP-complete - especially when considering hardware intricacies like micro-ops, hidden latencies, and kernel-specific overheads - these complexities are largely artifacts of our current approach to optimization.</p> <p>With full control over the execution schedule, we could eliminate many of these complexities. We would know exactly when each instruction will execute, making it trivial to determine the fastest solution among alternatives. We could precisely orchestrate the overlap of computation and communication, identify bottlenecks, and optimize the entire system's performance.</p> <p>The NP-completeness of this problem might seem daunting, but there are several factors that make it tractable in practice:</p> <ol> <li> <p><strong>Symmetrical Systems</strong>: Modern hardware is highly symmetrical. GPUs have uniform cores with identical L1 caches, evenly distributed L2 caches, and when using multiple nodes, they're typically equivalent. This symmetry means solutions for one subsystem often apply to the entire system, allowing us to break the problem into smaller, manageable chunks.</p> </li> <li> <p><strong>Simple Operations</strong>: The programs we're targeting - matrix multiplications, reductions, softmax operations - are relatively simple and have limited interactions. These operations are highly regular and don't have complex sequential dependencies, making them much easier to optimize than arbitrary Turing-complete programs.</p> </li> <li> <p><strong>Modular Solutions</strong>: The symmetrical nature of our target systems means we can often find solutions for parts of the system and apply them to similar components, reducing the overall complexity of the optimization problem.</p> </li> </ol> <p>It's important to note that this approach won't solve all problems. Complex programs with intricate sequential dependencies and arbitrary Turing-complete computations will still be challenging to optimize for distributed systems. However, for the specific class of problems we're targeting - large-scale numerical computations and machine learning workloads - this approach offers a promising path forward.</p> <h2 id="a-new-approach">A New Approach<a class="zola-anchor" href="#a-new-approach" aria-label="Anchor link for: a-new-approach"><i class="fas fa-link"></i></a> </h2> <p>Let's start with a simple example: summing the elements of a vector. In traditional Rust, we might write something like this:</p> <pre class="giallo z-code"><code data-lang="rust"><span class="giallo-l"><span class="z-keyword">fn</span><span class="z-entity z-name z-function"> sum_vector</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">v</span><span class="z-keyword z-operator">:</span><span class="z-keyword z-operator"> &amp;</span><span class="z-punctuation">[</span><span class="z-entity z-name z-type z-numeric z-rust">f32</span><span class="z-punctuation">]</span><span class="z-punctuation">)</span><span class="z-keyword z-operator"> -&gt;</span><span class="z-entity z-name z-type z-numeric z-rust"> f32</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-source"> v</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">iter</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">sum</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-punctuation">}</span></span></code></pre> <p>This function is concise but leaves many important questions unanswered: Where does the vector live? How many elements does it have? Should we use a GPU for this computation? What if we have multiple nodes available?</p> <p>Now, consider a version where we provide this information through type annotations:</p> <pre class="giallo z-code"><code data-lang="rust"><span class="giallo-l"><span class="z-keyword">fn</span><span class="z-entity z-name z-function"> sum_vector</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">v</span><span class="z-keyword z-operator">:</span><span class="z-keyword z-operator"> &amp;</span><span class="z-punctuation">[</span><span class="z-entity z-name z-type z-numeric z-rust">f32</span><span class="z-punctuation">;</span><span class="z-entity z-name z-type"> N</span><span class="z-punctuation">]</span><span class="z-keyword z-operator"> @</span><span class="z-entity z-name z-type"> Disk</span><span class="z-punctuation">)</span><span class="z-keyword z-operator"> -&gt;</span><span class="z-entity z-name z-type z-numeric z-rust"> f32</span><span class="z-keyword z-operator"> @</span><span class="z-constant z-other z-caps z-rust"> RAM</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-source"> v</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">iter</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">sum</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-punctuation">}</span></span></code></pre> <p>This annotated version tells the compiler crucial information:</p> <ul> <li>The vector has N elements</li> <li>The input data lives on disk</li> <li>The result should be stored in RAM</li> </ul> <p>With this information, the compiler can make intelligent decisions:</p> <ul> <li>For small N (e.g., 4 elements), it might choose a single CPU implementation</li> <li>For large N (e.g., 1 million elements), it might distribute the computation across GPUs or nodes</li> <li>If the input is on disk, it can plan the data movement accordingly</li> <li>It can generate multiple code paths optimized for different input sizes</li> </ul> <p>The key insight is that N doesn't need to be known at compile time. We could specify a range of possible values:</p> <pre class="giallo z-code"><code data-lang="rust"><span class="giallo-l"><span class="z-keyword">fn</span><span class="z-entity z-name z-function"> sum_vector</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">v</span><span class="z-keyword z-operator">:</span><span class="z-keyword z-operator"> &amp;</span><span class="z-punctuation">[</span><span class="z-entity z-name z-type z-numeric z-rust">f32</span><span class="z-punctuation">;</span><span class="z-entity z-name z-type"> N</span><span class="z-punctuation">]</span><span class="z-keyword z-operator"> @</span><span class="z-entity z-name z-type"> Disk</span><span class="z-punctuation">)</span><span class="z-keyword z-operator"> -&gt;</span><span class="z-entity z-name z-type z-numeric z-rust"> f32</span><span class="z-keyword z-operator"> @</span><span class="z-constant z-other z-caps z-rust"> RAM</span><span class="z-source"> </span></span> <span class="giallo-l"><span class="z-keyword">where</span></span> <span class="giallo-l"><span class="z-entity z-name z-type"> N</span><span class="z-keyword z-operator">:</span><span class="z-entity z-name z-type"> Range</span><span class="z-punctuation z-brackets z-angle z-rust">&lt;</span><span class="z-source">4</span><span class="z-punctuation">,</span><span class="z-source"> 1_000_000</span><span class="z-punctuation z-brackets z-angle z-rust">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation">{</span></span> <span class="giallo-l"><span class="z-source"> v</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">iter</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">sum</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-punctuation">}</span></span></code></pre> <p>The compiler can then:</p> <ol> <li>Analyze the performance characteristics for different N values</li> <li>Determine the breakpoints where different strategies become optimal</li> <li>Generate multiple code paths</li> <li>Choose the appropriate path at runtime based on the actual input size</li> </ol> <p>This approach allows us to:</p> <ul> <li>Make optimal decisions based on data characteristics</li> <li>Generate specialized code for different scenarios</li> <li>Handle data movement efficiently</li> <li>Adapt to the available hardware resources</li> </ul> <p>The compiler becomes an active participant in optimization, using the type information to make informed decisions about:</p> <ul> <li>Data distribution</li> <li>Computation strategy</li> <li>Memory management</li> <li>Hardware utilization</li> </ul> <h2 id="compiler-driven-optimization">Compiler-Driven Optimization<a class="zola-anchor" href="#compiler-driven-optimization" aria-label="Anchor link for: compiler-driven-optimization"><i class="fas fa-link"></i></a> </h2> <p>At first glance, one might worry that this approach would lead to verbose, annotation-laden code. After all, our example shows type annotations for data location, and it's natural to be concerned about this complexity spreading throughout the codebase. However, this isn't the case. Most functions in a program won't need these annotations at all, because we don't need to specify where intermediate computations happen.</p> <p>The annotations only appear at the boundaries of our computation - the entry points where data first enters our system (like reading from disk or receiving network data) and the exit points where we need the results in a specific location (like writing to a file or sending data over the network). These are the only places where we truly care about data location, as they represent the interfaces between our program and the outside world.</p> <p><img src="/assets/images/artisan_vs_factory.jpg" alt="Lazy Evaluation" /> <em>Figure 4: How the compiler manages data location decisions</em></p> <p>This approach allows the compiler to make intelligent decisions about data placement. For instance, in our vector sum example, if the data is already on the GPU from previous computations, the compiler can choose to keep it there rather than moving it to the CPU. The same function might execute on different hardware depending on the program's current state.</p> <p>The compiler handles the complexity of:</p> <ul> <li>Tracking where data lives throughout the program</li> <li>Making decisions about data placement</li> <li>Adapting to the current execution context</li> <li>Minimizing data movement</li> </ul> <h2 id="the-benefits">The Benefits<a class="zola-anchor" href="#the-benefits" aria-label="Anchor link for: the-benefits"><i class="fas fa-link"></i></a> </h2> <p>This approach offers several key advantages:</p> <ul> <li>Developers can focus on algorithms rather than implementation details</li> <li>The compiler can make global optimizations across the entire system</li> <li>The compiler can add Byzantine fault tolerance checks at system boundaries, detecting and adapting to nodes that are performing poorly or failing</li> <li>The compilation phase can predict exact execution times, enabling runtime verification of system behavior and detection of hardware issues</li> </ul> nicolas 2024-01-14T00:00:00+00:00 2024-01-14T00:00:00+00:00 https://nodata.dev/2024/01/14/tokenizerless/ <h1 id="what-tokenization-achieves">What tokenization achieves<a class="zola-anchor" href="#what-tokenization-achieves" aria-label="Anchor link for: what-tokenization-achieves"><i class="fas fa-link"></i></a> </h1> <p>Tokenization enables models to have some form of temporal compression (read outputting several characters as 1 timestep) while also increasing the entropy of those item. By definition we use compression algorithm to create the tokens, which means the distribution is trying to become closer the the uniform distribution.</p> <p>That means that models outputting random tokens will still make a coherent native looking output. It also means that the random distribution is a good starting point for the model itself (which wouldn't be the case with any real language)</p> <h1 id="tokenization-limits">Tokenization limits<a class="zola-anchor" href="#tokenization-limits" aria-label="Anchor link for: tokenization-limits"><i class="fas fa-link"></i></a> </h1> <p>Tokenization is limiting because it doesn't take into account previous tokens. Therefore, while the distribution of tokens is relatively uniform, the distribution of pairs of tokens is most definitely not, so some pairs are easier to train than others. This is what makes speculation working so well. One would think than having a more complex tokenizer would help here, but having more tokens in the vocabulary also is a cost, and because the tokenization methods are so crude, they yield diminishing returns.</p> <p>Also tokenization doesn't really work well with unicode. Unicode is the method to create all non ascii characters, where most non latin languages rely on. For a language like Chinese, you would need a vocabulary of size ~100k to just to get all the initial vocabulary (10k being a bare minimum). Because that's humongous and some of these bare characters are underrepresented, usually tokenizers remove some of these rare base chars, creating UNK tokens once again. The other technique is called byte fallback, meaning those unknown characters are shown to the model as their raw bytes representations. This works in training, but that also means that the model can output arbitrary sequences of raw bytes, meaning the model can also talk non valid utf-8 outputs which you have to deal with.</p> <h1 id="temporal-coherence">Temporal coherence<a class="zola-anchor" href="#temporal-coherence" aria-label="Anchor link for: temporal-coherence"><i class="fas fa-link"></i></a> </h1> <p>One thing to note, is that if you look at the output distribution of a transformers, even a simple one, there are definite breakpoints in <em>entropy</em>, and they really look like the original tokenizers, like "the ", or "or ".</p> <h1 id="multi-level-hierarchy">Multi level hierarchy<a class="zola-anchor" href="#multi-level-hierarchy" aria-label="Anchor link for: multi-level-hierarchy"><i class="fas fa-link"></i></a> </h1> <p>Achieving tokenizerless temporal coherence by pure ML ways, would enable models to stack various levels of temporal coherence on top of each other. There could be one level for the "tokens", but also some for the "sentences", or "paragraphs" or even "entire books".</p> <p>This is actually how most written content is written. Not exactly word for word in a continuous stream, but more a back and forth between multiple timescales. Write the abstract, expand some paragraphs, revisit the structure, rewrite sentences, fix typos, go back again to the top level structure, change a bit the conclusion, create a title, go back down to the first paragraph and so on. There is a constant jumping between timescale representations that produces modifications to another timescale representation.</p> <h1 id="results">Results<a class="zola-anchor" href="#results" aria-label="Anchor link for: results"><i class="fas fa-link"></i></a> </h1> nicolas 2020-07-29T00:00:00+00:00 2020-07-29T00:00:00+00:00 https://nodata.dev/2020/07/29/deploying-model-in-the-browser/ <blockquote> <p>TL;DR In this article I explain how you can deploy a model directly to the browser from pytorch by using Onnjx. This work was done a year ago in about two weeks time.</p> </blockquote> <p><img src="/assets/images/face/face.gif" alt="" /></p> <div style="width:100%;text-align:center;"> <a href="https://narsil.github.io/assets/face/">Check out the full demo</a>. </div> <h2 id="deploying-a-cool-deep-learning-demo-at-zero-cost">Deploying a cool deep learning demo at zero cost.<a class="zola-anchor" href="#deploying-a-cool-deep-learning-demo-at-zero-cost" aria-label="Anchor link for: deploying-a-cool-deep-learning-demo-at-zero-cost"><i class="fas fa-link"></i></a> </h2> <p>Ok, so when we are showcasing deep learning, usually that implies running models somewhere on the cloud. Sometimes, running these models is by itself quite costly. GPT-3 cost something like 10 million to train, but imagine how much it will cost to <em>run</em> if it was accessible to the general public !</p> <p>One technique applicable so small machine learning models, it to actually make to client run the model not you. This means that your front can be a simple static website. Hell you could even host it on Github for free !</p> <h2 id="background">Background<a class="zola-anchor" href="#background" aria-label="Anchor link for: background"><i class="fas fa-link"></i></a> </h2> <p>About a year ago, I was working at <a rel="nofollow external" href="https://nabla.com">Nabla</a> (They pivoted since). We were looking at how performant was 3d pose estimation of the face. It means models detecting faces <em>with</em> depth which was not as ubiquitous as regular 2d face detection.</p> <p>The idea was to see how hard it was to fit glasses on the fly to customers. The whole thing lasted for 2 weeks, so mind the lack of polish.</p> <h2 id="let-s-get-started-with-3ddfa">Let's get started with 3DDFA<a class="zola-anchor" href="#let-s-get-started-with-3ddfa" aria-label="Anchor link for: let-s-get-started-with-3ddfa"><i class="fas fa-link"></i></a> </h2> <p>So <a rel="nofollow external" href="https://github.com/cleardusk/3DDFA">3DDFA</a> is an improved Pytorch implementation of <a rel="nofollow external" href="https://arxiv.org/abs/1804.01005">this paper</a>. We settled on that implementation because it was the best available at the time.</p> <h3 id="how-does-it-work">How does it work ?<a class="zola-anchor" href="#how-does-it-work" aria-label="Anchor link for: how-does-it-work"><i class="fas fa-link"></i></a> </h3> <p>Most 3d pose estimation use the same tricks. First you need a good space representation with few latent variables. That is usually taken by creating a quite large dataset of a lot of people in lots of different poses and doing a very large PCA on this representation.</p> <p>For faces, 3ddfa uses a mix of <a rel="nofollow external" href="https://faces.dmi.unibas.ch/bfm/">BFM</a> and <a rel="nofollow external" href="http://kunzhou.net/2012/facewarehouse-tr.pdf">Facewarehouse</a>, for the full human body, <a rel="nofollow external" href="https://smpl.is.tue.mpg.de/">SMPL</a> is often used.</p> <p>In the face case, all 3D scans register the same 53490 vertices on the difference faces of the different participants (100 male and 100 female for BFM). For instance the center nose vertex:</p> <p><img src="/assets/images/face/bfm-vertex.png" alt="" /></p> <p>Then we have the same participants, with different expressions:</p> <p><img src="/assets/images/face/shape_expression_matrix.png" alt="" /></p> <p>Overall we end up with 150 participants x 20 poses x 53490 vertices (3 reals). We can then use a PCA to reduce the dimensionality into 2 orthogonal spaces, one for the <strong>shape</strong> and the other for the <strong>expression</strong> of a face. Such that any face can be expressed as.</p> <p>$$ M_{target\ face} = N_{neutral\ face} + \underbrace{S_{params} * S_{shape}}<em>{\text{Person identity}} + \underbrace{W</em>{params} * W_{expression}}_{\text{Smile or neutral expression}} $$</p> <p>Actually we're missing a global orientation of the face which can be represented as a rotation matrix and a scaling factor and an offset, which we will just express as a single matrix $ R $, and a vector $ p $.</p> <p>$$ M_{target\ face} = R \times ( N_{neutral\ face} + S_{params} * S_{shape} + W_{params} * W_{expression}) + p $$</p> <p>The 3DDFA model, will actually predict from a photo $ R $, $S_{params}$, $W_{params}$ and $p$. For reference, R is 9 floats, $S_{params}$ is 40 floats, $W_{params}$ is 10 floats, and $p$ is 3 floats. So the actual model of 3DDFA takes a 120x120 pixels image and returns a 62 vector that represents the face.</p> <p><img src="/assets/images/face/full-model.png" alt="" /></p> <p>The actual architecture of 3ddfa is actually just a simple mobile net.</p> <h2 id="let-s-port-the-model-to-the-browser">Let's port the model to the browser.<a class="zola-anchor" href="#let-s-port-the-model-to-the-browser" aria-label="Anchor link for: let-s-port-the-model-to-the-browser"><i class="fas fa-link"></i></a> </h2> <p>Ok, so let's isolate the first few lines in <code>main.py</code> that load the model.</p> <pre class="giallo z-code"><code data-lang="python"><span class="giallo-l"><span class="z-keyword">import</span><span class="z-source"> torch</span></span> <span class="giallo-l"><span class="z-keyword">import</span><span class="z-source"> mobilenet_v1</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment">#</span><span class="z-comment"> 1. load pre-tained model </span></span> <span class="giallo-l"><span class="z-source">checkpoint_fp</span><span class="z-keyword z-operator"> =</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">models/phase1_wpdc_vdc.pth.tar</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-source"> </span></span> <span class="giallo-l"><span class="z-source">arch</span><span class="z-keyword z-operator"> =</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">mobilenet_1</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-source"> </span></span> <span class="giallo-l"><span class="z-source"> </span></span> <span class="giallo-l"><span class="z-source">checkpoint</span><span class="z-keyword z-operator"> =</span><span class="z-source"> torch</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">load</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">checkpoint_fp</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> map_location</span><span class="z-keyword z-operator">=</span><span class="z-storage z-type">lambda</span><span class="z-variable z-parameter"> storage</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> loc</span><span class="z-punctuation">:</span><span class="z-meta z-function-call z-arguments z-python"> storage</span><span class="z-punctuation">)</span><span class="z-punctuation">[</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-string">state_dict</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">]</span></span> <span class="giallo-l"><span class="z-source">model</span><span class="z-keyword z-operator"> =</span><span class="z-support z-function z-builtin z-python"> getattr</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">mobilenet_v1</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> arch</span><span class="z-punctuation">)</span><span class="z-punctuation">(</span><span class="z-variable z-parameter">num_classes</span><span class="z-keyword z-operator">=</span><span class="z-constant z-numeric">62</span><span class="z-punctuation">)</span><span class="z-punctuation z-definition z-comment"> #</span><span class="z-comment"> 62 = 12(pose) + 40(shape) +10(expression)</span></span> <span class="giallo-l"><span class="z-source"> </span></span> <span class="giallo-l"><span class="z-source">model_dict</span><span class="z-keyword z-operator"> =</span><span class="z-source"> model</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">state_dict</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-source"> </span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment">#</span><span class="z-comment"> because the model is trained by multiple gpus, prefix module should be removed</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python">for</span><span class="z-source"> k</span><span class="z-keyword z-control z-flow z-python"> in</span><span class="z-source"> checkpoint</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">keys</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span><span class="z-source"> </span></span> <span class="giallo-l"><span class="z-meta z-indexed-name z-python"> model_dict</span><span class="z-punctuation">[</span><span class="z-meta z-item-access z-python">k</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">replace</span><span class="z-punctuation">(</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-string">module.</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">,</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">)</span><span class="z-punctuation">]</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-indexed-name z-python"> checkpoint</span><span class="z-punctuation">[</span><span class="z-meta z-item-access z-python">k</span><span class="z-punctuation">]</span><span class="z-source"> </span></span> <span class="giallo-l"><span class="z-source">model</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">load_state_dict</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">model_dict</span><span class="z-punctuation">)</span><span class="z-source"> </span></span></code></pre> <p>Now let's add a few lines to <a rel="nofollow external" href="https://pytorch.org/docs/stable/onnx.html">export the model to Onnx</a></p> <pre class="giallo z-code"><code data-lang="python"><span class="giallo-l"><span class="z-punctuation z-definition z-comment">#</span><span class="z-comment"> Batch size, C, H, W</span></span> <span class="giallo-l"><span class="z-source">dummy_input</span><span class="z-keyword z-operator"> =</span><span class="z-source"> torch</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">zeros</span><span class="z-punctuation">(</span><span class="z-punctuation">(</span><span class="z-constant z-numeric">1</span><span class="z-punctuation">,</span><span class="z-constant z-numeric"> 3</span><span class="z-punctuation">,</span><span class="z-constant z-numeric"> 120</span><span class="z-punctuation">,</span><span class="z-constant z-numeric"> 120</span><span class="z-punctuation">)</span><span class="z-punctuation"> )</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-source">torch</span><span class="z-punctuation">.</span><span class="z-source">onnx</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">export</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">model</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> dummy_input</span><span class="z-punctuation">,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">3ddfa.onnx</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> verbose</span><span class="z-keyword z-operator">=</span><span class="z-constant z-language z-python">True</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> input_names</span><span class="z-keyword z-operator">=</span><span class="z-punctuation">[</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">input</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-punctuation">]</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> output_names</span><span class="z-keyword z-operator">=</span><span class="z-punctuation">[</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">params</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-punctuation">]</span><span class="z-punctuation">)</span></span></code></pre> <p>Ok good, now we have a <code>3ddfa.onnx</code> file model in our directory.</p> <p>Let's try to run it in the browser by following <a rel="nofollow external" href="https://github.com/microsoft/onnxjs">OnnxJS</a> Getting Started and write a <code>index.html</code> file:</p> <pre class="giallo z-code"><code data-lang="html"><span class="giallo-l"><span class="z-punctuation z-definition z-tag">&lt;</span><span class="z-entity z-name z-tag">html</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-entity z-name z-tag">head</span><span class="z-punctuation z-definition z-tag">&gt;</span><span class="z-punctuation z-definition z-tag"> &lt;/</span><span class="z-entity z-name z-tag">head</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-entity z-name z-tag">body</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> &lt;!--</span><span class="z-comment"> Load ONNX.js </span><span class="z-punctuation z-definition z-comment">--&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-entity z-name z-tag">script</span><span class="z-entity z-other z-attribute-name"> src</span><span class="z-punctuation z-separator z-key-value">=</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">https://cdn.jsdelivr.net/npm/onnxjs/dist/onnx.min.js</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-punctuation z-definition z-tag">&gt;</span><span class="z-source">&lt;</span><span class="z-punctuation z-definition z-tag">/</span><span class="z-entity z-name z-tag">script</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> &lt;!--</span><span class="z-comment"> Code that consume ONNX.js </span><span class="z-punctuation z-definition z-comment">--&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-entity z-name z-tag">script</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> create a session</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> myOnnxSession</span><span class="z-keyword z-operator"> =</span><span class="z-keyword z-operator z-new"> new</span><span class="z-source"> onnx</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">InferenceSession</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> load the ONNX model file</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type"> function</span><span class="z-entity z-name z-function"> getInputs</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-punctuation">{</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> x</span><span class="z-keyword z-operator"> =</span><span class="z-keyword z-operator z-new"> new</span><span class="z-entity z-name z-function"> Float32Array</span><span class="z-source">(</span><span class="z-constant z-numeric">1</span><span class="z-keyword z-operator"> *</span><span class="z-constant z-numeric"> 3</span><span class="z-keyword z-operator"> *</span><span class="z-constant z-numeric"> 120</span><span class="z-keyword z-operator"> *</span><span class="z-constant z-numeric"> 120</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">fill</span><span class="z-source">(</span><span class="z-constant z-numeric">1</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> tensorX</span><span class="z-keyword z-operator"> =</span><span class="z-keyword z-operator z-new"> new</span><span class="z-source"> onnx</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">Tensor</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">x</span><span class="z-punctuation">,</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">float32</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">,</span><span class="z-source"> [</span><span class="z-constant z-numeric">1</span><span class="z-punctuation">,</span><span class="z-constant z-numeric"> 3</span><span class="z-punctuation">,</span><span class="z-constant z-numeric"> 120</span><span class="z-punctuation">,</span><span class="z-constant z-numeric"> 120</span><span class="z-source">]</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-keyword"> return</span><span class="z-source"> [</span><span class="z-variable z-other z-readwrite">tensorX</span><span class="z-source">]</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-source"> myOnnxSession</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">loadModel</span><span class="z-source">(</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">./3ddfa.onnx</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">then</span><span class="z-source">(</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-storage z-type"> =&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> generate model input</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> inferenceInputs</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-function"> getInputs</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> execute the model</span></span> <span class="giallo-l"><span class="z-source"> myOnnxSession</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">run</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">inferenceInputs</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">then</span><span class="z-source">(</span><span class="z-punctuation">(</span><span class="z-variable z-parameter">output</span><span class="z-punctuation">)</span><span class="z-storage z-type"> =&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> consume the output</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> outputTensor</span><span class="z-keyword z-operator"> =</span><span class="z-source"> output</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">values</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">next</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-js">value</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-source"> console</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">log</span><span class="z-source">(</span><span class="z-punctuation z-definition z-string">`</span><span class="z-string">model output tensor: </span><span class="z-punctuation">${</span><span class="z-variable z-other z-readwrite">outputTensor</span><span class="z-punctuation">}</span><span class="z-string">.</span><span class="z-punctuation z-definition z-string">`</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-source"> &lt;</span><span class="z-punctuation z-definition z-tag">/</span><span class="z-entity z-name z-tag">script</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;/</span><span class="z-entity z-name z-tag">body</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag">&lt;/</span><span class="z-entity z-name z-tag">html</span><span class="z-punctuation z-definition z-tag">&gt;</span></span></code></pre> <p>Now let's run a local server with <code>python -m http.server</code> and go to <code>http://localhost:8000</code> to see your console. Woops ! We got an error : <code>Uncaught (in promise) TypeError: cannot resolve operator 'Shape' with opsets: ai.onnx v9</code>.</p> <p>Actually onnx.js cannot infer dynamic shapes the same way PyTorch can. We can edit that by hardcoding some values within a <code>Reshape</code> (which <em>is</em> supported).</p> <p>In <code>mobilenet_v1.py</code> line 144, edit <code>x = x.view(x.shape(0), -1)</code> into <code>x.view(1, 1024)</code>. Now re-export your onnx file.</p> <p>It's working ! (Well we have values.)</p> <h2 id="putting-that-model-in-an-actual-demo-product">Putting that model in an actual demo product.<a class="zola-anchor" href="#putting-that-model-in-an-actual-demo-product" aria-label="Anchor link for: putting-that-model-in-an-actual-demo-product"><i class="fas fa-link"></i></a> </h2> <p>Ok so this is a bit more intensive work, so we're not going to detail every part, but focus on the most important parts.</p> <h3 id="get-the-webcam-data">Get the webcam data<a class="zola-anchor" href="#get-the-webcam-data" aria-label="Anchor link for: get-the-webcam-data"><i class="fas fa-link"></i></a> </h3> <pre class="giallo z-code"><code data-lang="javascript"><span class="giallo-l"><span class="z-storage z-type">var</span><span class="z-variable z-other z-readwrite"> that</span><span class="z-keyword z-operator"> =</span><span class="z-variable z-language z-this"> this</span></span> <span class="giallo-l"><span class="z-keyword">if</span><span class="z-source"> (</span><span class="z-source">navigator</span><span class="z-punctuation z-accessor">.</span><span class="z-source">mediaDevices</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-js">getUserMedia</span><span class="z-source">)</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-source"> navigator</span><span class="z-punctuation z-accessor">.</span><span class="z-source">mediaDevices</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">getUserMedia</span><span class="z-source">(</span><span class="z-punctuation">{</span><span class="z-source">video</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-punctuation"> {</span><span class="z-source">facingMode</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">user</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">}</span><span class="z-punctuation">}</span><span class="z-source">)</span></span> <span class="giallo-l"><span class="z-punctuation z-accessor"> .</span><span class="z-entity z-name z-function">then</span><span class="z-source">(</span><span class="z-storage z-type">function</span><span class="z-punctuation">(</span><span class="z-variable z-parameter">stream</span><span class="z-punctuation">)</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-source"> that</span><span class="z-punctuation z-accessor">.</span><span class="z-source">video</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-js">srcObject</span><span class="z-keyword z-operator"> =</span><span class="z-variable z-other z-readwrite"> stream</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-source"> that</span><span class="z-punctuation z-accessor">.</span><span class="z-source">video</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">play</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">then</span><span class="z-source">(</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-storage z-type"> =&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> That draws the video on a canvas.</span></span> <span class="giallo-l"><span class="z-source"> that</span><span class="z-punctuation z-accessor">.</span><span class="z-source">facedetector</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">loadModel</span><span class="z-source">(</span><span class="z-source">)</span></span> <span class="giallo-l"><span class="z-source"> that</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">loop</span><span class="z-source">(</span><span class="z-source">)</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">catch</span><span class="z-source">(</span><span class="z-punctuation">(</span><span class="z-variable z-parameter">e</span><span class="z-punctuation">)</span><span class="z-storage z-type"> =&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> alert</span><span class="z-source">(</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">Error launching webcam </span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword z-operator"> +</span><span class="z-variable z-other z-readwrite"> e</span><span class="z-source">)</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">catch</span><span class="z-source">(</span><span class="z-storage z-type">function</span><span class="z-punctuation">(</span><span class="z-variable z-parameter">e</span><span class="z-punctuation">)</span><span class="z-punctuation">{</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> alert</span><span class="z-source">(</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">No webcam detected </span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword z-operator"> +</span><span class="z-variable z-other z-readwrite"> e</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation">}</span></span></code></pre><h3 id="get-the-3d-faces-from-the-image">Get the 3d faces from the image<a class="zola-anchor" href="#get-the-3d-faces-from-the-image" aria-label="Anchor link for: get-the-3d-faces-from-the-image"><i class="fas fa-link"></i></a> </h3> <p>We're using <a rel="nofollow external" href="https://github.com/justadudewhohacks/face-api.js">face detector</a> to get where the faces are on the current image, then run our model on each face in the image.</p> <pre class="giallo z-code"><code data-lang="javascript"><span class="giallo-l"><span class="z-storage z-type">var</span><span class="z-variable z-other z-readwrite"> ctx</span><span class="z-keyword z-operator"> =</span><span class="z-source"> outcanvas</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">getContext</span><span class="z-source">(</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-string">2d</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-source">)</span><span class="z-keyword z-operator">!</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-storage z-type">const</span><span class="z-variable z-other z-constant z-js"> detections</span><span class="z-keyword z-operator"> =</span><span class="z-keyword"> await</span><span class="z-source"> faceapi</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">detectAllFaces</span><span class="z-source">(</span></span> <span class="giallo-l"><span class="z-variable z-other z-readwrite"> incanvas</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-keyword z-operator z-new"> new</span><span class="z-source"> faceapi</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">TinyFaceDetectorOptions</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type">var</span><span class="z-variable z-other z-readwrite"> vertices</span><span class="z-keyword z-operator"> =</span><span class="z-source"> [</span><span class="z-source">]</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-keyword">for</span><span class="z-source"> (</span><span class="z-variable z-other z-readwrite">i</span><span class="z-keyword z-operator"> =</span><span class="z-constant z-numeric"> 0</span><span class="z-punctuation">;</span><span class="z-variable z-other z-readwrite"> i</span><span class="z-keyword z-operator"> &lt;</span><span class="z-source"> detections</span><span class="z-punctuation z-accessor">.</span><span class="z-source">length</span><span class="z-punctuation">;</span><span class="z-variable z-other z-readwrite"> i</span><span class="z-keyword z-operator"> +=</span><span class="z-constant z-numeric"> 1</span><span class="z-source">)</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-storage z-type"> var</span><span class="z-variable z-other z-readwrite"> detection</span><span class="z-keyword z-operator"> =</span><span class="z-variable z-other z-readwrite"> detections</span><span class="z-source">[</span><span class="z-variable z-other z-readwrite">i</span><span class="z-source">]</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> We get a 1x3x120x120 Tensor here.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> We could batch that in theory, but simplicity here.</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> inferenceInputs</span><span class="z-keyword z-operator"> =</span><span class="z-variable z-language z-this"> this</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">getInputs</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">incanvas</span><span class="z-punctuation">,</span><span class="z-variable z-other z-readwrite"> detection</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> outputData</span><span class="z-keyword z-operator"> =</span><span class="z-keyword"> await</span><span class="z-variable z-language z-this"> this</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-js">session</span><span class="z-keyword z-operator">!</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">run</span><span class="z-source">(</span><span class="z-source">[</span><span class="z-variable z-other z-readwrite">inferenceInputs</span><span class="z-source">]</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> output</span><span class="z-keyword z-operator"> =</span><span class="z-source"> outputData</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">values</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">next</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-js">value</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> We need a reconstruction.</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> face_vertices</span><span class="z-keyword z-operator"> =</span><span class="z-variable z-language z-this"> this</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">reconstruct68</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">output</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-source"> </span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> Ellipsed code where we fuse various meshes to only run a single rotation</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> and render process (we need to render to occlude the glasses in 3d with</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> a transparent mesh of the face.</span></span> <span class="giallo-l"><span class="z-punctuation">}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-source">ctx</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">drawImage</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">incanvas</span><span class="z-punctuation">,</span><span class="z-constant z-numeric"> 0</span><span class="z-punctuation">,</span><span class="z-constant z-numeric"> 0</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment">//</span><span class="z-comment"> Drawing back the occluded glasses on the webcam canvas.</span></span> <span class="giallo-l"><span class="z-variable z-language z-this">this</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">drawGlasses</span><span class="z-source">(</span><span class="z-variable z-language z-this">this</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-js">scene</span><span class="z-punctuation">,</span><span class="z-variable z-other z-readwrite"> outcanvas</span><span class="z-source">)</span><span class="z-punctuation">;</span></span></code></pre><h3 id="reconstruct-the-vertices">Reconstruct the vertices.<a class="zola-anchor" href="#reconstruct-the-vertices" aria-label="Anchor link for: reconstruct-the-vertices"><i class="fas fa-link"></i></a> </h3> <p>As it turns out the Rotation ($R$) + Offset ($p$) we've seen in the first part is not really what it seems, $R$ is just a 3x3 matrix, nothing forces it to be rotation (meaning determinant 1). As it turns out the model, changes the determinant of this matrix quite heavily throughout time. Using it as is within our demo, would mean that glasses would grow large, then small and get deformed all the time.</p> <p>In order to fix this, we could fix the model of course, but it would require recreating a dataset, changing the loss functions and so on. But that's at least a full week to run that, and nothing guarantees that it would work. It could still fail because the morphable model lacks expressability, or you can't recreate the dataset because of some other issues and so on. That's of course the best course of action for a full featured product. But it won't ship fast.</p> <p>A much easier way is to recover the real rotation, scale and offset by running a small gradient descent loop locally, we know they exist because a real face does <em>not</em> change scale across time, rotation and offset tend to get small changes frame to frame.</p> <p>So what we're going to do, is reconstruct 68 vertices from the faces from the model. (we need more that 3 to stabilise the solution, 53k is way overboard) and <code>solve</code> the equation $$ \text{face vertices} = R \times s \times \text{glass vertices} + p $$.</p> <p>That's 13 params (9 + 1 + 3) for 68 equations. Inverting that system is not very practical as far as I know in javasript, so we're going to simply run a gradient descent. It should be faster than inversion on subsequent updates (when the solution is close to the previous solution).</p> <p>The full <a rel="nofollow external" href="https://github.com/Narsil/face/blob/master/src/solver.ts">code</a> is a bit scary looking (definitely not the way I would solve this now) but mostly amounts to manually working the gradient steps.</p> <h3 id="add-glasses-on-that-model">Add glasses on that model.<a class="zola-anchor" href="#add-glasses-on-that-model" aria-label="Anchor link for: add-glasses-on-that-model"><i class="fas fa-link"></i></a> </h3> <p>If we added the glasses mesh right away, we would have an occlusion problem where the back branch would show on top of your face. What we're going to do is to have a grouped mesh of the glasses <em>and</em> a transparent face (we'll use a mean normal face to keep computation low) so that the 3d rendering will occlude the back branch of the glasses.</p> <p>Adding the general face:</p> <pre class="giallo z-code"><code data-lang="javascript"><span class="giallo-l"><span class="z-variable z-other z-readwrite">public</span><span class="z-entity z-name z-function"> add_face</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">transparent</span><span class="z-keyword z-operator">?</span><span class="z-keyword z-operator">:</span><span class="z-variable z-other z-readwrite"> boolean</span><span class="z-source">)</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-keyword"> if</span><span class="z-source"> (</span><span class="z-variable z-other z-readwrite">transparent</span><span class="z-keyword z-operator"> ===</span><span class="z-constant z-language z-undefined z-js"> undefined</span><span class="z-source">)</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-variable z-other z-readwrite"> transparent</span><span class="z-keyword z-operator"> =</span><span class="z-constant z-language z-boolean"> true</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> self</span><span class="z-keyword z-operator"> =</span><span class="z-variable z-language z-this"> this</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-variable z-language z-this"> this</span><span class="z-punctuation z-accessor">.</span><span class="z-source">scene</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">load</span><span class="z-source">(</span><span class="z-source">process</span><span class="z-punctuation z-accessor">.</span><span class="z-source">env</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-constant">PUBLIC_URL</span><span class="z-keyword z-operator"> +</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">/3dmodels/face.fbx</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">then</span><span class="z-source">(</span><span class="z-variable z-parameter">face</span><span class="z-storage z-type"> =&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-keyword"> if</span><span class="z-source"> (</span><span class="z-variable z-other z-readwrite">transparent</span><span class="z-source">)</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> mesh</span><span class="z-keyword z-operator"> =</span><span class="z-source"> face</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-js">children</span><span class="z-source">[</span><span class="z-constant z-numeric">0</span><span class="z-source">]</span><span class="z-keyword"> as</span><span class="z-entity z-name z-type"> THREE</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-type">Mesh</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-source"> mesh</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-js">renderOrder</span><span class="z-keyword z-operator"> =</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">1</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-js"> material</span><span class="z-keyword z-operator"> =</span><span class="z-source"> mesh</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-js">material</span><span class="z-keyword"> as</span><span class="z-entity z-name z-type"> THREE</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-type">Material</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-source"> </span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> Makes the face occluding, but we write the background</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> Color, so alpha instead of texture.</span></span> <span class="giallo-l"><span class="z-source"> material</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-js">colorWrite</span><span class="z-keyword z-operator"> =</span><span class="z-constant z-language z-boolean"> false</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span></span> <span class="giallo-l"><span class="z-source"> self</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">add</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">face</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-source"> self</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">reset_clones</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation">}</span></span></code></pre> <p>Adding the glasses</p> <pre class="giallo z-code"><code data-lang="javascript"><span class="giallo-l"><span class="z-variable z-other z-readwrite">public</span><span class="z-entity z-name z-function"> addElement</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">element</span><span class="z-source">: </span><span class="z-variable z-other z-readwrite">string</span><span class="z-source">)</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-variable z-language z-this"> this</span><span class="z-punctuation z-accessor">.</span><span class="z-source">group</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">add_element</span><span class="z-source">(</span><span class="z-source">process</span><span class="z-punctuation z-accessor">.</span><span class="z-source">env</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-constant">PUBLIC_URL</span><span class="z-keyword z-operator"> +</span><span class="z-punctuation z-definition z-string"> `</span><span class="z-string">/3dmodels/</span><span class="z-punctuation">${</span><span class="z-variable z-other z-readwrite">element</span><span class="z-punctuation">}</span><span class="z-string">.fbx</span><span class="z-punctuation z-definition z-string">`</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation">}</span></span></code></pre> <p>The whole pipeline is a bit complex to reuse meshes and run the thing efficiently (still pretty messy code) but that's the gist, add the 2 meshes on the same scene, and hit render. Actually, we add all the faces and all the glasses to the same global scene and render only once.</p> <h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion"><i class="fas fa-link"></i></a> </h2> <p>So we've seen how we put a model from a working paper code, into an actual product. It necessary to remember to understand how the paper actually works vs pretends to work is important, it was super important here to understand the $R$ and $p$ and $s$ factors into our reconstruction. It also demonstrates that the paper's authors didn't realize that what they thought was a pose actually contains <em>a lot</em> of distortion, so it does not work how they expected.</p> <p>It also shows that putting ML in a product even for a very simple demo requires understanding many other technologies. Here it meant getting the webcam feed, getting Onnx layers simplfied, understanding <a rel="nofollow external" href="https://threejs.org/">Three.js</a> to master occlusion, etc... But by going full browser you can put your demo out and never worry about the cost !</p> <p>Check out the <a rel="nofollow external" href="https://github.com/Narsil/face">Full source code</a> for all the quirks.</p> nicolas 2020-07-27T00:00:00+00:00 2020-07-27T00:00:00+00:00 https://nodata.dev/ml/nlp/react/2020/07/27/creating-a-translation-app/ <blockquote> <p>TL;DR Recently moved to the Netherlands, in order to avoid Googling translate everything, I did the next best thing to learning the language: I created a clone of translate.google.com</p> </blockquote> <h2 id="find-a-correct-training-loop">Find a correct training loop<a class="zola-anchor" href="#find-a-correct-training-loop" aria-label="Anchor link for: find-a-correct-training-loop"><i class="fas fa-link"></i></a> </h2> <p>My first instinct was to check <a rel="nofollow external" href="https://github.com/huggingface/transformers">Hugging Face</a> as this repo contains solid implementations that I know are easy to change. However, in that particular instance, the example for translation does not start from scratch, and I wanted to check what multilingual translation could do here, as I'm using English, Dutch &amp; French on translate.google.com (For food sometimes french is much better than english for me).</p> <p>My second guess was <a rel="nofollow external" href="https://github.com/facebookresearch/fairseq">Fairseq</a> from facebook. In their example there is an actual example for multilingual German, French, English. Close enough for my needs. First things first, start to follow the example by the book. Most implementations out there are broken and won't work out of the box.</p> <p>This time, it turned out particularly smooth. Clone the repo then follow the <a rel="nofollow external" href="https://github.com/facebookresearch/fairseq/tree/main/examples/translation#multilingual-translation">instructions</a></p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span># First install sacrebleu and sentencepiece</span></span> <span class="giallo-l"><span>pip install sacrebleu sentencepiece</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span># Then download and preprocess the data</span></span> <span class="giallo-l"><span>cd examples/translation/</span></span> <span class="giallo-l"><span>bash prepare-iwslt17-multilingual.sh</span></span> <span class="giallo-l"><span>cd ../..</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span># Binarize the de-en dataset</span></span> <span class="giallo-l"><span>TEXT=examples/translation/iwslt17.de_fr.en.bpe16k</span></span> <span class="giallo-l"><span>fairseq-preprocess --source-lang de --target-lang en \</span></span> <span class="giallo-l"><span> --trainpref $TEXT/train.bpe.de-en \</span></span> <span class="giallo-l"><span> --validpref $TEXT/valid0.bpe.de-en,$TEXT/valid1.bpe.de-en,$TEXT/valid2.bpe.de-en,$TEXT/valid3.bpe.de-en,$TEXT/valid4.bpe.de-en,$TEXT/valid5.bpe.de-en \</span></span> <span class="giallo-l"><span> --destdir data-bin/iwslt17.de_fr.en.bpe16k \</span></span> <span class="giallo-l"><span> --workers 10</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span># Binarize the fr-en dataset</span></span> <span class="giallo-l"><span># NOTE: it&#39;s important to reuse the en dictionary from the previous step</span></span> <span class="giallo-l"><span>fairseq-preprocess --source-lang fr --target-lang en \</span></span> <span class="giallo-l"><span> --trainpref $TEXT/train.bpe.fr-en \</span></span> <span class="giallo-l"><span> --validpref $TEXT/valid0.bpe.fr-en,$TEXT/valid1.bpe.fr-en,$TEXT/valid2.bpe.fr-en,$TEXT/valid3.bpe.fr-en,$TEXT/valid4.bpe.fr-en,$TEXT/valid5.bpe.fr-en \</span></span> <span class="giallo-l"><span> --tgtdict data-bin/iwslt17.de_fr.en.bpe16k/dict.en.txt \</span></span> <span class="giallo-l"><span> --destdir data-bin/iwslt17.de_fr.en.bpe16k \</span></span> <span class="giallo-l"><span> --workers 10</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span># Train a multilingual transformer model</span></span> <span class="giallo-l"><span># NOTE: the command below assumes 1 GPU, but accumulates gradients from</span></span> <span class="giallo-l"><span># 8 fwd/bwd passes to simulate training on 8 GPUs</span></span> <span class="giallo-l"><span>mkdir -p checkpoints/multilingual_transformer</span></span> <span class="giallo-l"><span>CUDA_VISIBLE_DEVICES=0 fairseq-train data-bin/iwslt17.de_fr.en.bpe16k/ \</span></span> <span class="giallo-l"><span> --max-epoch 50 \</span></span> <span class="giallo-l"><span> --ddp-backend=no_c10d \</span></span> <span class="giallo-l"><span> --task multilingual_translation --lang-pairs de-en,fr-en \</span></span> <span class="giallo-l"><span> --arch multilingual_transformer_iwslt_de_en \</span></span> <span class="giallo-l"><span> --share-decoders --share-decoder-input-output-embed \</span></span> <span class="giallo-l"><span> --optimizer adam --adam-betas &#39;(0.9, 0.98)&#39; \</span></span> <span class="giallo-l"><span> --lr 0.0005 --lr-scheduler inverse_sqrt --min-lr &#39;1e-09&#39; \</span></span> <span class="giallo-l"><span> --warmup-updates 4000 --warmup-init-lr &#39;1e-07&#39; \</span></span> <span class="giallo-l"><span> --label-smoothing 0.1 --criterion label_smoothed_cross_entropy \</span></span> <span class="giallo-l"><span> --dropout 0.3 --weight-decay 0.0001 \</span></span> <span class="giallo-l"><span> --save-dir checkpoints/multilingual_transformer \</span></span> <span class="giallo-l"><span> --max-tokens 4000 \</span></span> <span class="giallo-l"><span> --update-freq 8</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span># Generate and score the test set with sacrebleu</span></span> <span class="giallo-l"><span>SRC=de</span></span> <span class="giallo-l"><span>sacrebleu --test-set iwslt17 --language-pair ${SRC}-en --echo src \</span></span> <span class="giallo-l"><span> | python scripts/spm_encode.py --model examples/translation/iwslt17.de_fr.en.bpe16k/sentencepiece.bpe.model \</span></span> <span class="giallo-l"><span> &gt; iwslt17.test.${SRC}-en.${SRC}.bpe</span></span> <span class="giallo-l"><span>cat iwslt17.test.${SRC}-en.${SRC}.bpe \</span></span> <span class="giallo-l"><span> | fairseq-interactive data-bin/iwslt17.de_fr.en.bpe16k/ \</span></span> <span class="giallo-l"><span> --task multilingual_translation --lang-pairs de-en,fr-en \</span></span> <span class="giallo-l"><span> --source-lang ${SRC} --target-lang en \</span></span> <span class="giallo-l"><span> --path checkpoints/multilingual_transformer/checkpoint_best.pt \</span></span> <span class="giallo-l"><span> --buffer-size 2000 --batch-size 128 \</span></span> <span class="giallo-l"><span> --beam 5 --remove-bpe=sentencepiece \</span></span> <span class="giallo-l"><span> &gt; iwslt17.test.${SRC}-en.en.sys</span></span></code></pre><h2 id="the-data">The data<a class="zola-anchor" href="#the-data" aria-label="Anchor link for: the-data"><i class="fas fa-link"></i></a> </h2> <p>While it's training, let's look at where I can get Dutch data. The IWSLT 2017 did not seem to have Dutch data <a rel="nofollow external" href="https://wit3.fbk.eu/mt.php?release=2017-01-trnted">at first glance</a> or <a rel="nofollow external" href="https://wit3.fbk.eu/mt.php?release=2017-01-trnmted">here</a>. I also tried just mimicking the adress from facebook <code>prepare-iwslt17-multilingual.sh</code> (The address <code>https://wit3.fbk.eu/archive/2017-01-trnted/texts/de/en/de-en.tgz</code> so I simply tried if <code>https://wit3.fbk.eu/archive/2017-01-trnted/texts/nl/en/nl-en.tgz</code>). Turns out there aren't. <a rel="nofollow external" href="https://www.statmt.org/europarl/">Europarl</a> seemed like a good bet but looking at the data, the langage seems pretty formatted and not very dialogue like. That might explain why it does not seem to be used that often. Looking back at IWSLT 2017 finally found the <a rel="nofollow external" href="https://wit3.fbk.eu/mt.php?release=2017-01-mted-test">Dutch data</a> and the <a rel="nofollow external" href="https://wit3.fbk.eu/mt.php?release=2017-01-trnmted">training data</a>. Is it me, or are competitions websites really hard to read ?</p> <h2 id="the-actual-training-loop">The actual training loop<a class="zola-anchor" href="#the-actual-training-loop" aria-label="Anchor link for: the-actual-training-loop"><i class="fas fa-link"></i></a> </h2> <p>Ok so let's reuse the training loop from the german file, so we just need to copy the dutch files in the same layout as the german ones, edit all the scripts and command lines to edit everything. I had to multiply the test files, someone Facebook has tst2011, tst2012 tst2013, tst2014, tst2015 for the german data, which does not seem to exist on the competition website... So here instead of trying to figure out where the information was, I simply copy-pasted the tst2010 file into dummy versions for tst2011...tst2015 (oh yeah simply omitting them will make bash scripts fail because file alignement is a requirement !, and I don't want to spend more than 5mn editing a bash script).</p> <p>Now with our edited bash script:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>cd examples/translation/</span></span> <span class="giallo-l"><span>bash prepare-iwslt17-multilingual_nl.sh</span></span> <span class="giallo-l"><span>cd ../..</span></span></code></pre> <p>Preprocess dutch data:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>TEXT=examples/translation/iwslt17.nl.en.bpe16k</span></span> <span class="giallo-l"><span>fairseq-preprocess --source-lang nl --target-lang en \</span></span> <span class="giallo-l"><span> --trainpref $TEXT/train.bpe.nl-en \</span></span> <span class="giallo-l"><span> --validpref $TEXT/valid0.bpe.nl-en,$TEXT/valid1.bpe.nl-en,$TEXT/valid2.bpe.nl-en,$TEXT/valid3.bpe.nl-en,$TEXT/valid4.bpe.nl-en,$TEXT/valid5.bpe.nl-en \</span></span> <span class="giallo-l"><span> --destdir data-bin/iwslt17.nl_fr.en.bpe16k \</span></span> <span class="giallo-l"><span> --workers 10</span></span></code></pre> <p>Now let's preprocess french data</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span># NOTE: it&#39;s important to reuse the en dictionary from the previous step</span></span> <span class="giallo-l"><span>fairseq-preprocess --source-lang fr --target-lang en \</span></span> <span class="giallo-l"><span> --trainpref $TEXT/train.bpe.fr-en \</span></span> <span class="giallo-l"><span> --validpref $TEXT/valid0.bpe.fr-en,$TEXT/valid1.bpe.fr-en,$TEXT/valid2.bpe.fr-en,$TEXT/valid3.bpe.fr-en,$TEXT/valid4.bpe.fr-en,$TEXT/valid5.bpe.fr-en \</span></span> <span class="giallo-l"><span> --tgtdict data-bin/iwslt17.nl_fr.en.bpe16k/dict.en.txt \</span></span> <span class="giallo-l"><span> --destdir data-bin/iwslt17.nl_fr.en.bpe16k \</span></span> <span class="giallo-l"><span> --workers 10</span></span></code></pre> <p>Overall, pretty simple task, just a bit bothering to hit all the various walls.</p> <p>Now that we preformatted the dutch data, we can run the training loop on our own data !</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>mkdir -p checkpoints/multilingual_transformer_nl</span></span> <span class="giallo-l"><span>CUDA_VISIBLE_DEVICES=0 fairseq-train data-bin/iwslt17.nl_fr.en.bpe16k/ \</span></span> <span class="giallo-l"><span> --max-epoch 50 \</span></span> <span class="giallo-l"><span> --ddp-backend=no_c10d \</span></span> <span class="giallo-l"><span> --task multilingual_translation --lang-pairs nl-en,fr-en \</span></span> <span class="giallo-l"><span> # Don&#39;t change the arch !\</span></span> <span class="giallo-l"><span> --arch multilingual_transformer_iwslt_de_en \</span></span> <span class="giallo-l"><span> --share-decoders --share-decoder-input-output-embed \</span></span> <span class="giallo-l"><span> --optimizer adam --adam-betas &#39;(0.9, 0.98)&#39; \</span></span> <span class="giallo-l"><span> --lr 0.0005 --lr-scheduler inverse_sqrt --min-lr &#39;1e-09&#39; \</span></span> <span class="giallo-l"><span> --warmup-updates 4000 --warmup-init-lr &#39;1e-07&#39; \</span></span> <span class="giallo-l"><span> --label-smoothing 0.1 --criterion label_smoothed_cross_entropy \</span></span> <span class="giallo-l"><span> --dropout 0.3 --weight-decay 0.0001 \</span></span> <span class="giallo-l"><span> # Change the checkpoint \</span></span> <span class="giallo-l"><span> --save-dir checkpoints/multilingual_transformer_nl \</span></span> <span class="giallo-l"><span> --max-tokens 4000 \</span></span> <span class="giallo-l"><span> --update-freq 8</span></span></code></pre><h2 id="checking-the-final-result">Checking the final result<a class="zola-anchor" href="#checking-the-final-result" aria-label="Anchor link for: checking-the-final-result"><i class="fas fa-link"></i></a> </h2> <p>So now we have a model <code>checkpoints/multilingual_transformer_nl/checkpoint_best.pt</code>, let's run it !</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span># Generate and score the test set with sacrebleu</span></span> <span class="giallo-l"><span>SRC=nl</span></span> <span class="giallo-l"><span>sacrebleu --test-set iwslt17 --language-pair ${SRC}-en --echo src \</span></span> <span class="giallo-l"><span> | python scripts/spm_encode.py --model examples/translation/iwslt17.nl_fr.en.bpe16k/sentencepiece.bpe.model \</span></span> <span class="giallo-l"><span> &gt; iwslt17.test.${SRC}-en.${SRC}.bpe</span></span> <span class="giallo-l"><span>cat iwslt17.test.${SRC}-en.${SRC}.bpe \</span></span> <span class="giallo-l"><span> | fairseq-interactive data-bin/iwslt17.nl_fr.en.bpe16k/ \</span></span> <span class="giallo-l"><span> --task multilingual_translation --lang-pairs de-en,fr-en \</span></span> <span class="giallo-l"><span> --source-lang ${SRC} --target-lang en \</span></span> <span class="giallo-l"><span> --path checkpoints/multilingual_transformer_nl/checkpoint_best.pt \</span></span> <span class="giallo-l"><span> --buffer-size 2000 --batch-size 128 \</span></span> <span class="giallo-l"><span> --beam 5 --remove-bpe=sentencepiece \</span></span> <span class="giallo-l"><span> &gt; iwslt17.test.${SRC}-en.en.sys</span></span></code></pre> <p>But woops...<code>sacreBLEU: No such language pair "nl-en" sacreBLEU: Available language pairs for test set "iwslt17": en-fr, fr-en, en-de, de-en, en-zh, zh-en, en-ar, ar-en, en-ja, ja-en, en-ko, ko-en</code></p> <p>So it looks like we're going to need to pipe some of our own data into this pipeline, we can just use the validation set we used to train</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>cat examples/translation/iwslt17.nl_fr.en.bpe16k/valid0.bpe.nl-en.nl |</span></span> <span class="giallo-l"><span>python scripts/spm_encode.py --model examples/translation/iwslt17.nl_fr.en.bpe16k/sentencepiece.bpe.model \</span></span> <span class="giallo-l"><span> &gt; iwslt17.test.${SRC}-en.${SRC}.bpe</span></span></code></pre> <p>There we go we have encoded with our multilingual BPE tokenizer our valid dataset. We can now run our translating command</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>cat iwslt17.test.${SRC}-en.${SRC}.bpe | fairseq-interactive data-bin/iwslt17.nl_fr.en.bpe16k/ --task multilingual_translation --lang-pairs nl-en,fr-en --source-lang ${SRC} --target-lang en --path checkpoints/multilingual_transformer_nl/checkpoint_best.pt --buffer-size 2000 --batch-size 128 --beam 5 --remove-bpe=sentencepiece</span></span></code></pre> <p>Here are some outputs (not cherry picked):</p> <pre class="giallo z-code"><code data-lang="rust"><span class="giallo-l"><span class="z-entity z-name z-type">S</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">999</span><span class="z-entity z-name z-type"> Iedereen</span><span class="z-source"> heeft</span><span class="z-source"> een</span><span class="z-source"> vissenkom</span><span class="z-source"> nodig</span><span class="z-keyword z-operator">.</span></span> <span class="giallo-l"><span class="z-entity z-name z-type">H</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">999</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">1</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">0272072553634644</span><span class="z-entity z-name z-type"> Everybody</span><span class="z-source"> needs</span><span class="z-source"> a</span><span class="z-source"> fishing</span><span class="z-source"> ticket</span><span class="z-keyword z-operator">.</span></span> <span class="giallo-l"><span class="z-entity z-name z-type">D</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">999</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">1</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">0272072553634644</span><span class="z-entity z-name z-type"> Everybody</span><span class="z-source"> needs</span><span class="z-source"> a</span><span class="z-source"> fishing</span><span class="z-source"> ticket</span><span class="z-keyword z-operator">.</span></span> <span class="giallo-l"><span class="z-entity z-name z-type">P</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">999</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">1</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">5687</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">2169</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">2363</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">2</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">0637</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">2</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">6527</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">2981</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">1540</span></span></code></pre><pre class="giallo z-code"><code data-lang="rust"><span class="giallo-l"><span class="z-entity z-name z-type">S</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">998</span><span class="z-entity z-name z-type"> Het</span><span class="z-source"> leidt</span><span class="z-source"> tot</span><span class="z-source"> meer</span><span class="z-source"> verlamming</span><span class="z-source"> en</span><span class="z-source"> minder</span><span class="z-source"> tevredenheid</span><span class="z-keyword z-operator">.</span></span> <span class="giallo-l"><span class="z-entity z-name z-type">H</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">998</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">32848915457725525</span><span class="z-entity z-name z-type"> It</span><span class="z-source"> leads</span><span class="z-source"> to</span><span class="z-source"> more</span><span class="z-source"> paralysis</span><span class="z-source"> and</span><span class="z-source"> less</span><span class="z-source"> satisfaction</span><span class="z-keyword z-operator">.</span></span> <span class="giallo-l"><span class="z-entity z-name z-type">D</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">998</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">32848915457725525</span><span class="z-entity z-name z-type"> It</span><span class="z-source"> leads</span><span class="z-source"> to</span><span class="z-source"> more</span><span class="z-source"> paralysis</span><span class="z-source"> and</span><span class="z-source"> less</span><span class="z-source"> satisfaction</span><span class="z-keyword z-operator">.</span></span> <span class="giallo-l"><span class="z-entity z-name z-type">P</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">998</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">9783</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">3836</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">1854</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">8328</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">1779</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">0163</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">3334</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">3619</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">2152</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">0450</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">2831</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">1289</span></span></code></pre><pre class="giallo z-code"><code data-lang="rust"><span class="giallo-l"><span class="z-entity z-name z-type">S</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">987</span><span class="z-entity z-name z-type"> Ze</span><span class="z-source"> maken</span><span class="z-source"> ons</span><span class="z-source"> leven</span><span class="z-source"> minder</span><span class="z-source"> waard</span><span class="z-keyword z-operator">.</span></span> <span class="giallo-l"><span class="z-entity z-name z-type">H</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">987</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">5473383665084839</span><span class="z-entity z-name z-type"> They</span><span class="z-source"> make</span><span class="z-source"> our</span><span class="z-source"> lives</span><span class="z-source"> worth</span><span class="z-source"> less</span><span class="z-keyword z-operator">.</span></span> <span class="giallo-l"><span class="z-entity z-name z-type">D</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">987</span><span class="z-keyword z-operator"> -</span><span class="z-constant z-numeric">0</span><span class="z-punctuation">.</span><span class="z-constant z-numeric">5473383665084839</span><span class="z-entity z-name z-type"> They</span><span class="z-source"> make</span><span class="z-source"> our</span><span class="z-source"> lives</span><span class="z-source"> worth</span><span class="z-source"> less</span><span class="z-keyword z-operator">.</span></span></code></pre> <p>Seems good enough for now.</p> <h2 id="productizing">Productizing<a class="zola-anchor" href="#productizing" aria-label="Anchor link for: productizing"><i class="fas fa-link"></i></a> </h2> <h3 id="flask-server">Flask server<a class="zola-anchor" href="#flask-server" aria-label="Anchor link for: flask-server"><i class="fas fa-link"></i></a> </h3> <p>Ok, in order to productionize, initially I wanted to move away from fairseq, but a lot of logic is actually tied to fairseq-interative (beam search, loading all the args, ensembling the model, source language selection and so on). It's definitely possible to move out of it, but it felt like a few days job, so much more than I was willing to invest in this particular approach.</p> <p>So the idea is to have a flask server sitting in front of the model, call the appropriate encoding with spm_encode, pass it to fairseq interactive, and output the D-XXX line back to the caller.</p> <p>We're going to containerize it and deploy to Kubernetes (it just happens I have a kubernetes cluster running, so less problems with deploying on it). I considered using ONNX-js (or TFlite) to deploy directly on the browser which saves a lot of headaches on deployment and keeping the service running in the long run (Like I did for the <a rel="nofollow external" href="https://narsil.github.io/assets/face/">glasses</a> project). Here the main problem is the size of the model (600Mo). I could go back and try to optimize but that's a pretty big model, it's going to be hard to make it come to a comfortable level for browser-only mode (Again just too much work for what I have in mind here).</p> <p>So let's get started from the Flask's <a rel="nofollow external" href="https://flask.palletsprojects.com/en/1.1.x/quickstart/">hello world</a></p> <pre class="giallo z-code"><code data-lang="python"><span class="giallo-l"><span class="z-keyword">from</span><span class="z-source"> flask</span><span class="z-keyword"> import</span><span class="z-source"> Flask</span></span> <span class="giallo-l"><span class="z-source">app</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> Flask</span><span class="z-punctuation">(</span><span class="z-support z-variable z-magic z-python">__name__</span><span class="z-punctuation">)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-decorator z-python">@</span><span class="z-entity z-name z-function z-decorator z-python">app</span><span class="z-punctuation">.</span><span class="z-entity z-name z-function z-decorator z-python">route</span><span class="z-punctuation">(</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-string">/</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python">def</span><span class="z-entity z-name z-function"> hello_world</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">Hello, World!</span><span class="z-punctuation z-definition z-string">&#39;</span></span></code></pre> <p>Let's edit it a bit to include our translate function.</p> <pre class="giallo z-code"><code data-lang="python"><span class="giallo-l"><span class="z-keyword">from</span><span class="z-source"> flask</span><span class="z-keyword"> import</span><span class="z-source"> Flask</span></span> <span class="giallo-l"><span class="z-source">app</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> Flask</span><span class="z-punctuation">(</span><span class="z-support z-variable z-magic z-python">__name__</span><span class="z-punctuation">)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python">def</span><span class="z-entity z-name z-function"> translate</span><span class="z-punctuation">(</span><span class="z-variable z-parameter">text</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> #</span><span class="z-keyword"> TODO</span><span class="z-comment"> later</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">This is a translation !</span><span class="z-punctuation z-definition z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-decorator z-python">@</span><span class="z-entity z-name z-function z-decorator z-python">app</span><span class="z-punctuation">.</span><span class="z-entity z-name z-function z-decorator z-python">route</span><span class="z-punctuation">(</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-string">/</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> methods</span><span class="z-keyword z-operator">=</span><span class="z-punctuation">[</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">POST</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-punctuation">]</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python">def</span><span class="z-entity z-name z-function"> hello</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-source"> text</span><span class="z-keyword z-operator"> =</span><span class="z-source"> request</span><span class="z-punctuation">.</span><span class="z-meta z-indexed-name z-python">form</span><span class="z-punctuation">[</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">input</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-punctuation">]</span></span> <span class="giallo-l"><span class="z-support z-function z-builtin z-python"> print</span><span class="z-punctuation">(</span><span class="z-storage z-type z-string z-python">f</span><span class="z-string">&quot;</span><span class="z-string">IN </span><span class="z-constant z-character z-format z-placeholder z-other z-python">{</span><span class="z-meta z-function-call z-arguments z-python">text</span><span class="z-constant z-character z-format z-placeholder z-other z-python">}</span><span class="z-string">&quot;</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-source"> output</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> translate</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">text</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-support z-function z-builtin z-python"> print</span><span class="z-punctuation">(</span><span class="z-storage z-type z-string z-python">f</span><span class="z-string">&quot;</span><span class="z-string">OUT </span><span class="z-constant z-character z-format z-placeholder z-other z-python">{</span><span class="z-meta z-function-call z-arguments z-python">output</span><span class="z-constant z-character z-format z-placeholder z-other z-python">}</span><span class="z-string">&quot;</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-source"> result</span><span class="z-keyword z-operator"> =</span><span class="z-source"> json</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">dumps</span><span class="z-punctuation">(</span><span class="z-punctuation">{</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">en</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-punctuation">:</span><span class="z-meta z-function-call z-arguments z-python"> output</span><span class="z-punctuation">}</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-source"> result</span></span></code></pre> <p>We can run our example and check it's running with curl</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>$ curl -d input=&quot;Ik heft een appel.&quot; http://localhost:5000/`</span></span> <span class="giallo-l"><span>{&quot;en&quot;: &quot;This is a translation !&quot;}</span></span></code></pre><h3 id="implementing-the-translate-function">Implementing the translate function.<a class="zola-anchor" href="#implementing-the-translate-function" aria-label="Anchor link for: implementing-the-translate-function"><i class="fas fa-link"></i></a> </h3> <p>Ok this is where we are super tied to fairseq-interactive code, I had to dig into the source code, copy most of it, and mainly split <code>Model loading</code> code from <code>Model running</code> code. For that I used a lot of globals as the original code does not separate these two concerns (tidying this will be a later goal if it every comes to that).</p> <p>The final implementation is quite verbose but available <a rel="nofollow external" href="https://github.com/Narsil/translate/blob/master/server/translate.py">here</a>.</p> <p>One good point about this implementation is that we load the model early, so that it's available right away when the server comes up (but it does take some time to come up). A negative point, is that because it's loaded eagerly it's going to make forking a nightmare and basically preventing us from using wsgi efficiently which is the <a rel="nofollow external" href="https://flask.palletsprojects.com/en/1.1.x/deploying/">recommended way of deploying Flask</a>. It's fine for now, it's a personnal project after all, to get more stable deployment I would try to remove python from the equation of the web part if possible, it's really slow and hard to work with on webservers because of the forking/threading nightmare in Python.</p> <p>So know our backend can really translate !</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>$ curl -d input=&quot;Ik heft een appel.&quot; http://localhost:5000/`</span></span> <span class="giallo-l"><span>{&quot;en&quot;: &quot;I have an apple.&quot;}</span></span></code></pre> <p>Before moving that to the cloud, let's build a nice interface in front of it</p> <h3 id="react-front">React front<a class="zola-anchor" href="#react-front" aria-label="Anchor link for: react-front"><i class="fas fa-link"></i></a> </h3> <p>Ok so we're going to use React with Typescript. React because we're going JS anyway to get the translation without clicking a button with a form like html. It's also more convenient to use Material-UI which I find helps make a website nice from scratch (and I'm tired of Bootstrap). Typescript because it's just saner than VanillaJS (it won't make much of a difference here).</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>yarn create react-app app --template typescript</span></span> <span class="giallo-l"><span>cd app</span></span> <span class="giallo-l"><span>yarn add @material-ui/core</span></span></code></pre> <p>Let's edit our App.tsx to use Material-UI and get the initial layout looking like <a href="https://nodata.dev/ml/nlp/react/2020/07/27/creating-a-translation-app/translate.google.com">translate.google.com</a>.</p> <pre class="giallo z-code"><code data-lang="tsx"><span class="giallo-l"><span class="z-keyword">import</span><span class="z-variable z-other z-readwrite"> React</span><span class="z-keyword"> from</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">react</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-keyword">import</span><span class="z-punctuation"> {</span><span class="z-variable z-other z-readwrite">makeStyles</span><span class="z-punctuation">}</span><span class="z-keyword"> from</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">@material-ui/core/styles</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-keyword">import</span><span class="z-variable z-other z-readwrite"> TextField</span><span class="z-keyword"> from</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">@material-ui/core/TextField</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-keyword">import</span><span class="z-variable z-other z-readwrite"> Card</span><span class="z-keyword"> from</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">@material-ui/core/Card</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-keyword">import</span><span class="z-variable z-other z-readwrite"> Grid</span><span class="z-keyword"> from</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">@material-ui/core/Grid</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type">const</span><span class="z-variable z-other z-constant"> useStyles</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-function"> makeStyles</span><span class="z-source">(</span><span class="z-variable z-parameter">theme</span><span class="z-storage z-type"> =&gt;</span><span class="z-source"> (</span><span class="z-punctuation">{</span></span> <span class="giallo-l"><span class="z-source"> app</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-source"> display</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">flex</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-source"> justifyContent</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">center</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-source"> alignItems</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">center</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-source"> height</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">100vh</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation">}</span><span class="z-source">)</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type">function</span><span class="z-entity z-name z-function"> App</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant"> classes</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-function"> useStyles</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword"> return</span><span class="z-source"> (</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-entity z-name z-tag">div</span><span class="z-entity z-other z-attribute-name"> className</span><span class="z-keyword z-operator">=</span><span class="z-punctuation">{</span><span class="z-source">classes</span><span class="z-punctuation z-accessor">.</span><span class="z-source">app</span><span class="z-punctuation">}</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-support z-class z-component z-tsx">Card</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-entity z-name z-tag">form</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-support z-class z-component z-tsx">Grid</span><span class="z-entity z-other z-attribute-name"> container</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-support z-class z-component z-tsx">Grid</span><span class="z-entity z-other z-attribute-name"> item</span><span class="z-entity z-other z-attribute-name"> xs</span><span class="z-keyword z-operator">=</span><span class="z-punctuation">{</span><span class="z-constant z-numeric">12</span><span class="z-punctuation">}</span><span class="z-entity z-other z-attribute-name"> md</span><span class="z-keyword z-operator">=</span><span class="z-punctuation">{</span><span class="z-constant z-numeric">6</span><span class="z-punctuation">}</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-support z-class z-component z-tsx">TextField</span></span> <span class="giallo-l"><span class="z-entity z-other z-attribute-name"> id</span><span class="z-keyword z-operator">=</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">standard-basic</span><span class="z-punctuation z-definition z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-other z-attribute-name"> label</span><span class="z-keyword z-operator">=</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">Dutch</span><span class="z-punctuation z-definition z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-other z-attribute-name"> multiline</span></span> <span class="giallo-l"><span class="z-entity z-other z-attribute-name"> autoFocus</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> /&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;/</span><span class="z-support z-class z-component z-tsx">Grid</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-support z-class z-component z-tsx">Grid</span><span class="z-entity z-other z-attribute-name"> item</span><span class="z-entity z-other z-attribute-name"> xs</span><span class="z-keyword z-operator">=</span><span class="z-punctuation">{</span><span class="z-constant z-numeric">12</span><span class="z-punctuation">}</span><span class="z-entity z-other z-attribute-name"> md</span><span class="z-keyword z-operator">=</span><span class="z-punctuation">{</span><span class="z-constant z-numeric">6</span><span class="z-punctuation">}</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;</span><span class="z-support z-class z-component z-tsx">TextField</span><span class="z-entity z-other z-attribute-name"> id</span><span class="z-keyword z-operator">=</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">standard-basic</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-entity z-other z-attribute-name"> label</span><span class="z-keyword z-operator">=</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">English</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-entity z-other z-attribute-name"> multiline</span><span class="z-punctuation z-definition z-tag"> /&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;/</span><span class="z-support z-class z-component z-tsx">Grid</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;/</span><span class="z-support z-class z-component z-tsx">Grid</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;/</span><span class="z-entity z-name z-tag">form</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;/</span><span class="z-support z-class z-component z-tsx">Card</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-tag"> &lt;/</span><span class="z-entity z-name z-tag">div</span><span class="z-punctuation z-definition z-tag">&gt;</span></span> <span class="giallo-l"><span class="z-source"> )</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation">}</span></span> <span class="giallo-l"><span class="z-keyword">export</span><span class="z-keyword"> default</span><span class="z-variable z-other z-readwrite"> App</span><span class="z-punctuation">;</span></span></code></pre> <p>Here is the result : <img src="https://i.imgur.com/ZszCVQU.png" alt="" /></p> <p>Now let's look at the logic (simplified):</p> <pre class="giallo z-code"><code data-lang="typescript"><span class="giallo-l"><span class="z-storage z-type">type</span><span class="z-entity z-name z-type"> Timeout</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-type"> ReturnType</span><span class="z-punctuation z-definition z-typeparameters">&lt;</span><span class="z-keyword z-operator z-expression z-typeof z-ts">typeof</span><span class="z-variable z-other z-readwrite"> setTimeout</span><span class="z-punctuation z-definition z-typeparameters">&gt;</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type">const</span><span class="z-punctuation"> [</span><span class="z-variable z-other z-constant z-ts">text</span><span class="z-punctuation">,</span><span class="z-variable z-other z-constant z-ts"> setText</span><span class="z-punctuation">]</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-function"> useState</span><span class="z-source">(</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-storage z-type">const</span><span class="z-punctuation"> [</span><span class="z-variable z-other z-constant z-ts">time</span><span class="z-punctuation">,</span><span class="z-variable z-other z-constant z-ts"> setTime</span><span class="z-punctuation">]</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-function"> useState</span><span class="z-punctuation z-definition z-typeparameters">&lt;</span><span class="z-entity z-name z-type">Timeout</span><span class="z-keyword z-operator"> |</span><span class="z-support z-type z-builtin z-ts"> null</span><span class="z-punctuation z-definition z-typeparameters">&gt;</span><span class="z-source">(</span><span class="z-constant z-language z-null z-ts">null</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-storage z-type">const</span><span class="z-variable z-other z-constant z-ts"> url</span><span class="z-keyword z-operator"> =</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">http://localhost:5000</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type">const</span><span class="z-entity z-name z-function"> translate</span><span class="z-keyword z-operator"> =</span><span class="z-punctuation"> (</span><span class="z-variable z-parameter">text</span><span class="z-keyword z-operator">:</span><span class="z-support z-type z-primitive"> string</span><span class="z-punctuation">)</span><span class="z-storage z-type z-function z-arrow z-ts"> =&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-keyword"> if</span><span class="z-source"> (</span><span class="z-variable z-other z-readwrite">text</span><span class="z-keyword z-operator"> ===</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-source">)</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> setText</span><span class="z-source">(</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-keyword"> return</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-ts"> form</span><span class="z-keyword z-operator"> =</span><span class="z-keyword z-operator z-new"> new</span><span class="z-entity z-name z-function"> FormData</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-source"> form</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">append</span><span class="z-source">(</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-string">input</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">,</span><span class="z-variable z-other z-readwrite"> text</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> fetch</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">url</span><span class="z-punctuation">,</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-source"> method</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">POST</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-source"> body</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-variable z-other z-readwrite"> form</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">then</span><span class="z-source">(</span><span class="z-variable z-parameter">response</span><span class="z-storage z-type z-function z-arrow z-ts"> =&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-source"> response</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">json</span><span class="z-source">(</span><span class="z-source">)</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">then</span><span class="z-source">(</span><span class="z-variable z-parameter">json</span><span class="z-storage z-type z-function z-arrow z-ts"> =&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-source"> console</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">log</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">json</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> setText</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">json</span><span class="z-source">[</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-string">en</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-source">]</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation">}</span><span class="z-punctuation">;</span></span></code></pre> <p>Then call it on the <code>onChange</code> attribute of our Dutch field.</p> <pre class="giallo z-code"><code data-lang="typescript"><span class="giallo-l"><span class="z-variable z-other z-readwrite">onChange</span><span class="z-keyword z-operator">=</span><span class="z-punctuation">{</span><span class="z-variable z-parameter">event</span><span class="z-storage z-type z-function z-arrow z-ts"> =&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> We use a timeout handler to prevent very fast keystrokes</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> //</span><span class="z-comment"> from spamming our server.</span></span> <span class="giallo-l"><span class="z-keyword"> if</span><span class="z-source"> (</span><span class="z-variable z-other z-readwrite">time</span><span class="z-keyword z-operator"> !==</span><span class="z-constant z-language z-null z-ts"> null</span><span class="z-source">)</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> clearTimeout</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">time</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-ts"> text</span><span class="z-keyword z-operator"> =</span><span class="z-source"> event</span><span class="z-punctuation z-accessor">.</span><span class="z-source">target</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-property z-ts">value</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-storage z-type"> const</span><span class="z-variable z-other z-constant z-ts"> timeout</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-function"> setTimeout</span><span class="z-source">(</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span><span class="z-storage z-type z-function z-arrow z-ts"> =&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> translate</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">text</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-punctuation">,</span><span class="z-constant z-numeric"> 500</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> setTime</span><span class="z-source">(</span><span class="z-variable z-other z-readwrite">timeout</span><span class="z-source">)</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation">}</span><span class="z-punctuation">}</span></span></code></pre> <p>There we have it:</p> <p><img src="https://i.imgur.com/EYZ0EWR.gif" alt="" /></p> <h3 id="let-s-dockerize">Let's dockerize !<a class="zola-anchor" href="#let-s-dockerize" aria-label="Anchor link for: let-s-dockerize"><i class="fas fa-link"></i></a> </h3> <p>As I mentionned loading the whole model in the flask app is going to hinder a lot the wsgi process forking. I did try it, try to come up with easy fixes, but ultimately found that keeping the development server was just easier.</p> <p>Ok so we're going to need a python docker image, install pytorch, fairseq, and flask to our image (actually we need flask_cors too to make sure we can call from any website as it's an API.)</p> <p>As it turns out, fairseq 0.9 had a bug in the training loop and I was using master from a few month ago, and I needed to work with that specific version since there had been breaking changes since in master. That gives us the following <code>requirements.txt</code></p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>torch</span></span> <span class="giallo-l"><span>flask</span></span> <span class="giallo-l"><span>flask_cors</span></span> <span class="giallo-l"><span>-e git://github.com/pytorch/fairseq.git@7a6519f84fed06947bbf161c7b66c9099bc4ce53#egg=fairseq</span></span> <span class="giallo-l"><span>sentencepiece</span></span></code></pre> <p>Now our Docker file, is going to get the python dependencies, copy all the local files (including model and tokenizer file) and run the flask server. That gives us :</p> <pre class="giallo z-code"><code data-lang="docker"><span class="giallo-l"><span class="z-keyword">FROM</span><span class="z-source"> python:3.7-slim</span></span> <span class="giallo-l"><span class="z-keyword">RUN</span><span class="z-source"> pip install -U pip</span></span> <span class="giallo-l"><span class="z-keyword">RUN</span><span class="z-source"> apt-get update &amp;&amp; apt-get install -y git build-essential # Required for building fairseq from source.</span></span> <span class="giallo-l"><span class="z-keyword">COPY</span><span class="z-source"> server/requirements.txt /app/requirements.txt</span></span> <span class="giallo-l"><span class="z-keyword">RUN</span><span class="z-source"> pip install -r /app/requirements.txt</span></span> <span class="giallo-l"><span class="z-keyword">COPY</span><span class="z-source"> . /app</span></span> <span class="giallo-l"><span class="z-keyword">WORKDIR</span><span class="z-source"> /app</span></span> <span class="giallo-l"><span class="z-keyword">CMD</span><span class="z-source"> [</span><span class="z-string">&quot;</span><span class="z-string">python</span><span class="z-string">&quot;</span><span class="z-source">, </span><span class="z-string">&quot;</span><span class="z-string">translate.py</span><span class="z-string">&quot;</span><span class="z-source">]</span></span></code></pre> <p>Let's build and check that it works:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>docker build -t translate:latest .</span></span> <span class="giallo-l"><span>docker run -p 5000:5000 translate:latest</span></span> <span class="giallo-l"><span># Now check with curl that we can still hit the docker and get a correct answer</span></span> <span class="giallo-l"><span>curl -d input=&quot;Ik heft een appel.&quot; http://localhost:5000/`</span></span> <span class="giallo-l"><span># {&quot;en&quot;: &quot;This is a translation !&quot;}</span></span></code></pre><h3 id="kubernetes-cluster">Kubernetes cluster<a class="zola-anchor" href="#kubernetes-cluster" aria-label="Anchor link for: kubernetes-cluster"><i class="fas fa-link"></i></a> </h3> <p>Okay the following part will be pretty specific to my setup. I use a kubernetes cluster on GCP with ingress. I'm going to skip updating the SSL certificate.</p> <p>Let's start with pushing the image to GCP:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>docker tag translate:latest gcr.io/myproject-XXXXXX/translate:1.0</span></span> <span class="giallo-l"><span>docker push gcr.io/myproject-XXXXXX/translate:1.0</span></span> <span class="giallo-l"><span>kubectl apply -f deployment.yaml</span></span> <span class="giallo-l"><span>kubectl apply -f service.yaml</span></span> <span class="giallo-l"><span>kubectl apply -f ingress.yaml</span></span> <span class="giallo-l"></span></code></pre> <p>Here are the (edited for brevity&amp;security) service files I used:</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment">#</span><span class="z-comment">deployment.yaml</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">a</span><span class="z-entity z-name z-tag z-yaml">piVersion</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> a</span><span class="z-string">pps/v1</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">k</span><span class="z-entity z-name z-tag z-yaml">ind</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> D</span><span class="z-string">eployment</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">m</span><span class="z-entity z-name z-tag z-yaml">etadata</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> n</span><span class="z-entity z-name z-tag z-yaml">ame</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> t</span><span class="z-string">ranslate-deployment</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">s</span><span class="z-entity z-name z-tag z-yaml">pec</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> r</span><span class="z-entity z-name z-tag z-yaml">eplicas</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-constant z-numeric"> 1</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> s</span><span class="z-entity z-name z-tag z-yaml">elector</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> m</span><span class="z-entity z-name z-tag z-yaml">atchLabels</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> a</span><span class="z-entity z-name z-tag z-yaml">pp</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> t</span><span class="z-string">ranslate</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> t</span><span class="z-entity z-name z-tag z-yaml">emplate</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> m</span><span class="z-entity z-name z-tag z-yaml">etadata</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> l</span><span class="z-entity z-name z-tag z-yaml">abels</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> a</span><span class="z-entity z-name z-tag z-yaml">pp</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> t</span><span class="z-string">ranslate</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> s</span><span class="z-entity z-name z-tag z-yaml">pec</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> c</span><span class="z-entity z-name z-tag z-yaml">ontainers</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-punctuation"> -</span><span class="z-entity z-name z-tag z-yaml"> n</span><span class="z-entity z-name z-tag z-yaml">ame</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> t</span><span class="z-string">ranslate</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> i</span><span class="z-entity z-name z-tag z-yaml">mage</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> g</span><span class="z-string">cr.io/myproject-XXXXX/translate:1.0</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> p</span><span class="z-entity z-name z-tag z-yaml">orts</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-punctuation"> -</span><span class="z-entity z-name z-tag z-yaml"> c</span><span class="z-entity z-name z-tag z-yaml">ontainerPort</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-constant z-numeric"> 5000</span></span></code></pre><pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment">#</span><span class="z-comment"> service.yaml</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">a</span><span class="z-entity z-name z-tag z-yaml">piVersion</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> v</span><span class="z-string">1</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">k</span><span class="z-entity z-name z-tag z-yaml">ind</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> S</span><span class="z-string">ervice</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">m</span><span class="z-entity z-name z-tag z-yaml">etadata</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> n</span><span class="z-entity z-name z-tag z-yaml">ame</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> t</span><span class="z-string">ranslate-service</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">s</span><span class="z-entity z-name z-tag z-yaml">pec</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> t</span><span class="z-entity z-name z-tag z-yaml">ype</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> N</span><span class="z-string">odePort</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> s</span><span class="z-entity z-name z-tag z-yaml">elector</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> a</span><span class="z-entity z-name z-tag z-yaml">pp</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> t</span><span class="z-string">ranslate</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> p</span><span class="z-entity z-name z-tag z-yaml">orts</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-punctuation"> -</span><span class="z-entity z-name z-tag z-yaml"> p</span><span class="z-entity z-name z-tag z-yaml">rotocol</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> T</span><span class="z-string">CP</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> p</span><span class="z-entity z-name z-tag z-yaml">ort</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-constant z-numeric"> 80</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> t</span><span class="z-entity z-name z-tag z-yaml">argetPort</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-constant z-numeric"> 5000</span></span></code></pre><pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment">#</span><span class="z-comment">ingress.yaml</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">a</span><span class="z-entity z-name z-tag z-yaml">piVersion</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> n</span><span class="z-string">etworking.k8s.io/v1beta1</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">k</span><span class="z-entity z-name z-tag z-yaml">ind</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> I</span><span class="z-string">ngress</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">m</span><span class="z-entity z-name z-tag z-yaml">etadata</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> n</span><span class="z-entity z-name z-tag z-yaml">ame</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> i</span><span class="z-string">ngress-front</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> a</span><span class="z-entity z-name z-tag z-yaml">nnotations</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> k</span><span class="z-entity z-name z-tag z-yaml">ubernetes.io/ingress.global-static-ip-name</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> a</span><span class="z-string">ddress-cluster</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> n</span><span class="z-entity z-name z-tag z-yaml">etworking.gke.io/managed-certificates</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> o</span><span class="z-string">ttomate-certificate-new</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml">s</span><span class="z-entity z-name z-tag z-yaml">pec</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> r</span><span class="z-entity z-name z-tag z-yaml">ules</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-punctuation"> -</span><span class="z-entity z-name z-tag z-yaml"> h</span><span class="z-entity z-name z-tag z-yaml">ost</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> t</span><span class="z-string">ranslate.ottomate.app</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> h</span><span class="z-entity z-name z-tag z-yaml">ttp</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> p</span><span class="z-entity z-name z-tag z-yaml">aths</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-punctuation"> -</span><span class="z-entity z-name z-tag z-yaml"> p</span><span class="z-entity z-name z-tag z-yaml">ath</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> /</span><span class="z-string">*</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> b</span><span class="z-entity z-name z-tag z-yaml">ackend</span><span class="z-punctuation z-separator z-key-value">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> s</span><span class="z-entity z-name z-tag z-yaml">erviceName</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-string"> t</span><span class="z-string">ranslate-service</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag z-yaml"> s</span><span class="z-entity z-name z-tag z-yaml">ervicePort</span><span class="z-punctuation z-separator z-key-value">:</span><span class="z-constant z-numeric"> 80</span></span></code></pre> <p>Hopefully within a few minutes you have your pod running and you can hit your live own server with the API.</p> <p>You just need to update your react App to point the the correct URL and boom your done, your very own translate server app.</p> <h3 id="what-could-should-be-done-next">What could/should be done next.<a class="zola-anchor" href="#what-could-should-be-done-next" aria-label="Anchor link for: what-could-should-be-done-next"><i class="fas fa-link"></i></a> </h3> <h4 id="for-the-model">For the model:<a class="zola-anchor" href="#for-the-model" aria-label="Anchor link for: for-the-model"><i class="fas fa-link"></i></a> </h4> <ul> <li>Add more data to the original training set, some words are missing, translation can become funky on some real world sentences I give the machine (Dutch companies tend to send very verbose emails)</li> <li>Add some data augmentation in the pool as the current translation is very brittle to errors. Using Sentence piece algorihm with sampling instead of BPE could be used, some typo generator, word inversions to name a few. Training some error detection algorithm on top or using ready made ones could help (translate.google.com has some spellfixing magic applied before it seems.)</li> <li>Making it smaller to make it portable to tflite, mobile phone for offline mode and so on (it's a pretty big workload to make it work though)</li> </ul> <h4 id="for-the-backend">For the backend:<a class="zola-anchor" href="#for-the-backend" aria-label="Anchor link for: for-the-backend"><i class="fas fa-link"></i></a> </h4> <ul> <li>Battle testing the backend should be the first thing to do to check failure modes and fix naive DOS attacks.</li> <li>Something like <a rel="nofollow external" href="https://github.com/pytorch/serve">TorchServe</a> seems like what we want for the model part. Never used it so far, but it seems to solve some problems encountered here and would make iterations faster on various models (also swapping out models).</li> <li>On the other spectrum I could go for tighter control. Removing the fairseq-interative clutter would be my first move. If I can go pytorch barebones, then using Rust, with Hugging Face's <a rel="nofollow external" href="https://github.com/huggingface/tokenizers">tokenizers</a> library would probably make inference faster and deployment easier. It would of course make iteration much slower so I would do that only when the model is very stable. It could make mobile offline possible (with a very large app data, ~1Go, but doable.)</li> </ul> <h4 id="for-the-frontend">For the frontend:<a class="zola-anchor" href="#for-the-frontend" aria-label="Anchor link for: for-the-frontend"><i class="fas fa-link"></i></a> </h4> <ul> <li>Working a bit more on the mobile part of the design which is a bit broken at the moment.</li> <li>Maybe add buttons to switch languages easily, switch language sides (although I mostly use Dutch-&gt;English and Dutch-&gt;French)</li> <li>Add a react-native app so that I can translate from my phone. (Without offline mode)</li> </ul> 2020-03-19T00:00:00+00:00 2020-03-19T00:00:00+00:00 https://nodata.dev/energy/2020/03/19/solar-energy/ <!-- ################################################# ### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ### ################################################# # file to edit: _notebooks/2020-03-19-solar-energy.ipynb --> <div class="container" id="notebook-container"> <div class="cell border-box-sizing code_cell rendered"> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h1 id="Solar-energy">Solar energy<a class="anchor-link" href="#Solar-energy"> </a></h1><h2 id="Stefan-boltzmann's-law">Stefan boltzmann's law<a class="anchor-link" href="#Stefan-boltzmann's-law"> </a></h2><p>$ \text{Surface energy} = \sigma T^4$</p> <p>For the sun, $T = \text{5,778 }K$</p> <p>$\sigma = 5.67 \times 10 ^{-8} W.m^{-2}.K^{-4}$</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">sympy.physics.units</span> <span class="kn">import</span> <span class="n">K</span><span class="p">,</span> <span class="n">W</span><span class="p">,</span> <span class="n">m</span><span class="p">,</span> <span class="n">giga</span> <span class="n">sigma</span> <span class="o">=</span> <span class="mf">5.67</span> <span class="o">*</span> <span class="mi">10</span><span class="o">**</span><span class="p">(</span><span class="o">-</span><span class="mi">8</span><span class="p">)</span> <span class="o">*</span> <span class="n">W</span> <span class="o">*</span><span class="n">m</span><span class="o">**</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span> <span class="o">*</span> <span class="n">K</span><span class="o">**</span><span class="p">(</span><span class="o">-</span><span class="mi">4</span><span class="p">)</span> <span class="n">T</span> <span class="o">=</span> <span class="mi">5778</span> <span class="o">*</span> <span class="n">K</span> <span class="n">surface_energy</span> <span class="o">=</span> <span class="n">sigma</span> <span class="o">*</span> <span class="n">T</span><span class="o">**</span><span class="mi">4</span> <span class="nb">print</span><span class="p">(</span><span class="n">surface_energy</span><span class="p">)</span> </pre></div> </div> </div> </div> <div class="output_wrapper"> <div class="output"> <div class="output_area"> <div class="output_subarea output_stream output_stdout output_text"> <pre>63196526.5460292*watt/meter**2 </pre> </div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="Total-emitted-solar-energy">Total emitted solar energy<a class="anchor-link" href="#Total-emitted-solar-energy"> </a></h2><p>$ Radiation = \text{Surface of the sun} \times \text{Surface energy} $</p> <p>$ Radiation = 4 \pi r^2 \times \text{Surface energy} $</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">sympy</span> <span class="kn">import</span> <span class="o">*</span> <span class="n">r_sun</span> <span class="o">=</span> <span class="mi">696_340</span> <span class="o">*</span> <span class="mi">1000</span> <span class="o">*</span><span class="n">m</span> <span class="n">surface_of_sun</span> <span class="o">=</span> <span class="mi">4</span> <span class="o">*</span> <span class="n">pi</span> <span class="o">*</span> <span class="n">r_sun</span> <span class="o">**</span> <span class="mi">2</span> <span class="n">radiation</span> <span class="o">=</span> <span class="n">surface_of_sun</span> <span class="o">*</span> <span class="n">surface_energy</span> <span class="nb">print</span><span class="p">(</span><span class="n">radiation</span><span class="p">)</span> </pre></div> </div> </div> </div> <div class="output_wrapper"> <div class="output"> <div class="output_area"> <div class="output_subarea output_stream output_stdout output_text"> <pre>1.22573302243694e+26*pi*watt </pre> </div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="Energy-received-at-earth-average-distance">Energy received at earth average distance<a class="anchor-link" href="#Energy-received-at-earth-average-distance"> </a></h2><p>$ \text{Radiation received} = \frac{\text{Total sun radiation}}{ \text{sphere at earth's distance}}$</p> <p>$ \text{Radiation received} = \frac{Radiation}{ 4 \pi D_{earth-sun}^2} $</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="n">R_earth</span> <span class="o">=</span> <span class="mi">6_371</span> <span class="o">*</span> <span class="mi">1000</span> <span class="o">*</span> <span class="n">m</span> <span class="n">D_earth_sun</span> <span class="o">=</span> <span class="mf">148.88</span> <span class="o">*</span> <span class="mi">10</span><span class="o">**</span><span class="mi">6</span> <span class="o">*</span> <span class="mi">1000</span> <span class="o">*</span> <span class="n">m</span> <span class="n">earth_perp_surface</span> <span class="o">=</span> <span class="n">pi</span> <span class="o">*</span> <span class="n">R_earth</span> <span class="o">**</span><span class="mi">2</span> <span class="n">sphere</span> <span class="o">=</span> <span class="mi">4</span> <span class="o">*</span> <span class="n">pi</span> <span class="o">*</span> <span class="n">D_earth_sun</span> <span class="o">**</span><span class="mi">2</span> <span class="n">radiation_received</span> <span class="o">=</span> <span class="n">radiation</span> <span class="o">/</span> <span class="n">sphere</span> <span class="nb">print</span><span class="p">(</span><span class="n">radiation_received</span><span class="p">)</span> </pre></div> </div> </div> </div> <div class="output_wrapper"> <div class="output"> <div class="output_area"> <div class="output_subarea output_stream output_stdout output_text"> <pre>1382.49374484614*watt/meter**2 </pre> </div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="Energy-received-by-the-earth-surface-(before-atmosphere)">Energy received by the earth surface (before atmosphere)<a class="anchor-link" href="#Energy-received-by-the-earth-surface-(before-atmosphere)"> </a></h2><p>$ \text{Energy received} = \text{radiation received} \times \frac{ \text{visible surface}}{ \text{earth's surface}} $</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="n">power_received</span> <span class="o">=</span> <span class="n">radiation_received</span> <span class="o">*</span> <span class="n">pi</span> <span class="o">*</span> <span class="n">R_earth</span> <span class="o">**</span><span class="mi">2</span> <span class="n">surface_power_received</span> <span class="o">=</span> <span class="n">power_received</span> <span class="o">/</span> <span class="p">(</span><span class="mi">4</span> <span class="o">*</span> <span class="n">pi</span> <span class="o">*</span> <span class="n">R_earth</span> <span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="n">surface_power_received</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="n">power_received</span><span class="o">.</span><span class="n">n</span><span class="p">())</span> </pre></div> </div> </div> </div> <div class="output_wrapper"> <div class="output"> <div class="output_area"> <div class="output_subarea output_stream output_stdout output_text"> <pre>345.623436211536*watt/meter**2 1.76290235470883e+17*watt </pre> </div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <blockquote><p>RADIATION RECEIVED BY SYSTEM EARTH = $345 W.m^{-2}$</p> <p>MAXIMUM POWER WITH EARTH "DYSON SPHERE": $176 PW$</p> </blockquote> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h1 id="Human-consumption">Human consumption<a class="anchor-link" href="#Human-consumption"> </a></h1><p>13 511 MTep <a href="https://www.iea.org/data-and-statistics?country=WORLD&amp;fuel=Energy%20supply&amp;indicator=Total%20primary%20energy%20supply%20%28TPES%29%20by%20source">Source International Energy agency</a></p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">sympy.physics.units</span> <span class="kn">import</span> <span class="n">J</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="n">W</span> <span class="kn">from</span> <span class="nn">sympy.physics.units.util</span> <span class="kn">import</span> <span class="n">convert_to</span> <span class="n">million</span> <span class="o">=</span> <span class="mi">10</span> <span class="o">**</span><span class="mi">6</span> <span class="n">kilo</span> <span class="o">=</span> <span class="mi">10</span><span class="o">**</span><span class="mi">3</span> <span class="n">giga</span> <span class="o">=</span> <span class="mi">10</span> <span class="o">**</span> <span class="mi">9</span> <span class="n">toe</span> <span class="o">=</span> <span class="mf">41.868</span> <span class="o">*</span> <span class="n">giga</span> <span class="o">*</span> <span class="n">J</span> <span class="n">ktoe</span> <span class="o">=</span> <span class="n">kilo</span> <span class="o">*</span> <span class="n">toe</span> <span class="n">Mtoe</span> <span class="o">=</span> <span class="n">million</span> <span class="o">*</span> <span class="n">toe</span> <span class="n">hour</span> <span class="o">=</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="n">s</span> <span class="n">year</span> <span class="o">=</span> <span class="mi">24</span> <span class="o">*</span> <span class="n">h</span> <span class="o">*</span> <span class="mf">365.25</span> <span class="n">base</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">([</span><span class="mi">3852538</span><span class="p">,</span><span class="mi">2949909</span><span class="p">,</span><span class="mi">670298</span><span class="p">,</span><span class="mi">335519</span><span class="p">,</span><span class="mi">204190</span><span class="p">,</span><span class="mi">1286064</span><span class="p">,</span><span class="mi">4329220</span><span class="p">])</span> <span class="n">Humanity_total_annual_consumption</span> <span class="o">=</span> <span class="n">base</span> <span class="o">*</span> <span class="n">ktoe</span> <span class="n">humanity_power_consumption</span> <span class="o">=</span> <span class="n">Humanity_total_annual_consumption</span> <span class="o">/</span> <span class="n">year</span> <span class="nb">print</span><span class="p">(</span><span class="n">convert_to</span><span class="p">(</span><span class="n">humanity_power_consumption</span><span class="o">.</span><span class="n">n</span><span class="p">(),</span> <span class="p">[</span><span class="n">W</span><span class="p">])</span><span class="o">.</span><span class="n">n</span><span class="p">())</span> </pre></div> </div> </div> </div> <div class="output_wrapper"> <div class="output"> <div class="output_area"> <div class="output_subarea output_stream output_stdout output_text"> <pre>18080149776408.9*watt </pre> </div> </div> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">convert_to</span><span class="p">(</span><span class="n">humanity_power_consumption</span> <span class="o">/</span> <span class="n">power_received</span> <span class="o">*</span> <span class="mi">100</span><span class="p">,</span> <span class="p">[</span><span class="n">J</span><span class="p">,</span> <span class="n">s</span><span class="p">])</span><span class="o">.</span><span class="n">n</span><span class="p">())</span> </pre></div> </div> </div> </div> <div class="output_wrapper"> <div class="output"> <div class="output_area"> <div class="output_subarea output_stream output_stdout output_text"> <pre>0.0102558997258785 </pre> </div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>We are currently consuming <strong>0.01% of the maximum capacity of the earth covered by a Dyson sphere of solar panels</strong>.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h3 id="A-bit-more-realistic-approach">A bit more realistic approach<a class="anchor-link" href="#A-bit-more-realistic-approach"> </a></h3> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>After the atmosphere only $168 W.m^{-2}$ hit the surface. It's quite complicated to infer it depends on the wavelength of the incoming light, clouds, composition of the atmosphere and so on, so we just take the value from <a href="https://fr.wikipedia.org/wiki/Bilan_radiatif_de_la_Terre">here</a>.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Then we only have 29% of the earth surface that is landmass (where we can reasonably put solar panels in large quantity)</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Of that 31% is covered in forest which are already some natural solar panels we don't want to remove (for other obvious reasons) <a href="http://www.earth-policy.org/indicators/C56/forests_2012">source</a> And 38.4% is covered of agricultural land <a href="https://en.wikipedia.org/wiki/Agricultural_land">source</a>.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Then solar panels are not 100% efficient. They are roughly only 20% efficient with current technology at a reasonable cost.</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="n">earth_power_received</span> <span class="o">=</span> <span class="mi">168</span> <span class="o">*</span> <span class="n">W</span> <span class="o">*</span> <span class="n">m</span> <span class="o">**</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span> <span class="n">available_surface</span> <span class="o">=</span> <span class="mi">4</span> <span class="o">*</span> <span class="n">pi</span> <span class="o">*</span> <span class="n">R_earth</span> <span class="o">**</span><span class="mi">2</span> <span class="o">*</span> <span class="mf">0.29</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-.</span><span class="mi">31</span> <span class="o">-</span> <span class="o">.</span><span class="mi">384</span><span class="p">)</span> <span class="n">max_power</span> <span class="o">=</span> <span class="n">earth_power_received</span> <span class="o">*</span> <span class="n">available_surface</span> <span class="o">*</span> <span class="mf">0.2</span> <span class="nb">print</span><span class="p">(</span><span class="n">max_power</span><span class="o">.</span><span class="n">n</span><span class="p">())</span> <span class="nb">print</span><span class="p">(</span><span class="n">convert_to</span><span class="p">(</span><span class="n">humanity_power_consumption</span> <span class="o">/</span> <span class="n">max_power</span> <span class="o">*</span><span class="mi">100</span><span class="p">,</span> <span class="p">[</span><span class="n">J</span><span class="p">,</span> <span class="n">s</span><span class="p">])</span><span class="o">.</span><span class="n">n</span><span class="p">())</span> </pre></div> </div> </div> </div> <div class="output_wrapper"> <div class="output"> <div class="output_area"> <div class="output_subarea output_stream output_stdout output_text"> <pre>1.52084087357243e+15*watt 1.18882587196246 </pre> </div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h1 id="Conclusion">Conclusion<a class="anchor-link" href="#Conclusion"> </a></h1> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>In the end we are currently consuming <strong>1.2% of the realistic available solar power energy</strong>. That's would require posing solar panels everywhere on the planet that is not a forest or agricultural land. And we don't account yet for Energy return on energy invested (EROEI) which is likely to increase that percentage.</p> <p>NB: This is a very superficial attempt to evaluate these numbers, however the result should be correct within an order of magnitude.</p> </div> </div> </div> </div> 2020-03-10T00:00:00+00:00 2020-03-10T00:00:00+00:00 https://nodata.dev/ml/2020/03/10/no-gd-training/ <!-- ################################################# ### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ### ################################################# # file to edit: _notebooks/2020-03-10-no-gd-training.ipynb --> <div class="container" id="notebook-container"> <div class="cell border-box-sizing code_cell rendered"> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="What's-the-problem-?">What's the problem ?<a class="anchor-link" href="#What's-the-problem-?"> </a></h2><p>ML models usually are not really capable of predicting how well the data you<br> feed them is close to what was in the dataset. It really matters in production models as they might make really stupid mistakes just because they are off<br> the training set.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Let's train a simple mnist model (straight out from pytorch tutorial <a href="https://github.com/pytorch/examples/tree/master/mnist">https://github.com/pytorch/examples/tree/master/mnist</a>)</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <details class="description"> <summary class="btn btn-sm" data-open="Hide Code" data-close="Show Code"></summary> <p><div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#collapse</span> <span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span> <span class="kn">import</span> <span class="nn">argparse</span> <span class="kn">import</span> <span class="nn">torch</span> <span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="nn">nn</span> <span class="kn">import</span> <span class="nn">torch.nn.functional</span> <span class="k">as</span> <span class="nn">F</span> <span class="kn">import</span> <span class="nn">torch.optim</span> <span class="k">as</span> <span class="nn">optim</span> <span class="kn">from</span> <span class="nn">torchvision</span> <span class="kn">import</span> <span class="n">datasets</span><span class="p">,</span> <span class="n">transforms</span> <span class="kn">from</span> <span class="nn">torch.optim.lr_scheduler</span> <span class="kn">import</span> <span class="n">StepLR</span> <span class="kn">import</span> <span class="nn">os</span> <span class="k">class</span> <span class="nc">Net</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="nb">super</span><span class="p">(</span><span class="n">Net</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">conv1</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Conv2d</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">conv2</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Conv2d</span><span class="p">(</span><span class="mi">32</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout1</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Dropout2d</span><span class="p">(</span><span class="mf">0.25</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout2</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Dropout2d</span><span class="p">(</span><span class="mf">0.5</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">fc1</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="mi">9216</span><span class="p">,</span> <span class="mi">128</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">fc2</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conv1</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conv2</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">max_pool2d</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout1</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">flatten</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">fc1</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout2</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">fc2</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">output</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">log_softmax</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="k">return</span> <span class="n">output</span> <span class="k">def</span> <span class="nf">train</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">train_loader</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">,</span> <span class="n">epoch</span><span class="p">):</span> <span class="n">model</span><span class="o">.</span><span class="n">train</span><span class="p">()</span> <span class="k">for</span> <span class="n">batch_idx</span><span class="p">,</span> <span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">train_loader</span><span class="p">):</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">target</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">zero_grad</span><span class="p">()</span> <span class="n">output</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="n">loss</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">nll_loss</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span> <span class="n">loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="k">if</span> <span class="n">batch_idx</span> <span class="o">%</span> <span class="n">args</span><span class="o">.</span><span class="n">log_interval</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Train Epoch: </span><span class="si">{}</span><span class="s1"> [</span><span class="si">{}</span><span class="s1">/</span><span class="si">{}</span><span class="s1"> (</span><span class="si">{:.0f}</span><span class="s1">%)]</span><span class="se">\t</span><span class="s1">Loss: </span><span class="si">{:.6f}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">epoch</span><span class="p">,</span> <span class="n">batch_idx</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">train_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">),</span> <span class="mf">100.</span> <span class="o">*</span> <span class="n">batch_idx</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">train_loader</span><span class="p">),</span> <span class="n">loss</span><span class="o">.</span><span class="n">item</span><span class="p">()))</span> <span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">):</span> <span class="n">model</span><span class="o">.</span><span class="n">eval</span><span class="p">()</span> <span class="n">test_loss</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">correct</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">with</span> <span class="n">torch</span><span class="o">.</span><span class="n">no_grad</span><span class="p">():</span> <span class="k">for</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="ow">in</span> <span class="n">test_loader</span><span class="p">:</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">target</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">output</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="n">test_loss</span> <span class="o">+=</span> <span class="n">F</span><span class="o">.</span><span class="n">nll_loss</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;sum&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="c1"># sum up batch loss</span> <span class="n">pred</span> <span class="o">=</span> <span class="n">output</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdim</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="c1"># get the index of the max log-probability</span> <span class="n">correct</span> <span class="o">+=</span> <span class="n">pred</span><span class="o">.</span><span class="n">eq</span><span class="p">(</span><span class="n">target</span><span class="o">.</span><span class="n">view_as</span><span class="p">(</span><span class="n">pred</span><span class="p">))</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="n">test_loss</span> <span class="o">/=</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\n</span><span class="s1">Test set: Average loss: </span><span class="si">{:.4f}</span><span class="s1">, Accuracy: </span><span class="si">{}</span><span class="s1">/</span><span class="si">{}</span><span class="s1"> (</span><span class="si">{:.0f}</span><span class="s1">%)</span><span class="se">\n</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">test_loss</span><span class="p">,</span> <span class="n">correct</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">),</span> <span class="mf">100.</span> <span class="o">*</span> <span class="n">correct</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">)))</span> <span class="k">def</span> <span class="nf">mnist</span><span class="p">():</span> <span class="n">filename</span> <span class="o">=</span><span class="s2">&quot;mnist_cnn.pt&quot;</span> <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span> <span class="k">return</span> <span class="c1"># Training settings</span> <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="s1">&#39;PyTorch MNIST Example&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--batch-size&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">64</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;input batch size for training (default: 64)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--test-batch-size&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;input batch size for testing (default: 1000)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--epochs&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;number of epochs to train (default: 14)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--lr&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;LR&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;learning rate (default: 1.0)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--gamma&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">0.7</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;M&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Learning rate step gamma (default: 0.7)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--no-cuda&#39;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s1">&#39;store_true&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;disables CUDA training&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--seed&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;S&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;random seed (default: 1)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--log-interval&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;how many batches to wait before logging training status&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--save-model&#39;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s1">&#39;store_true&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;For Saving the current Model&#39;</span><span class="p">)</span> <span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span> <span class="n">use_cuda</span> <span class="o">=</span> <span class="ow">not</span> <span class="n">args</span><span class="o">.</span><span class="n">no_cuda</span> <span class="ow">and</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="n">torch</span><span class="o">.</span><span class="n">manual_seed</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">seed</span><span class="p">)</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s2">&quot;cuda&quot;</span> <span class="k">if</span> <span class="n">use_cuda</span> <span class="k">else</span> <span class="s2">&quot;cpu&quot;</span><span class="p">)</span> <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;num_workers&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;pin_memory&#39;</span><span class="p">:</span> <span class="kc">True</span><span class="p">}</span> <span class="k">if</span> <span class="n">use_cuda</span> <span class="k">else</span> <span class="p">{}</span> <span class="n">train_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">datasets</span><span class="o">.</span><span class="n">MNIST</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="n">test_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">datasets</span><span class="o">.</span><span class="n">MNIST</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">expanduser</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">),</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">test_batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="n">model</span> <span class="o">=</span> <span class="n">Net</span><span class="p">()</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">optimizer</span> <span class="o">=</span> <span class="n">optim</span><span class="o">.</span><span class="n">Adadelta</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">parameters</span><span class="p">(),</span> <span class="n">lr</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">lr</span><span class="p">)</span> <span class="n">scheduler</span> <span class="o">=</span> <span class="n">StepLR</span><span class="p">(</span><span class="n">optimizer</span><span class="p">,</span> <span class="n">step_size</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">gamma</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">gamma</span><span class="p">)</span> <span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">epochs</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span> <span class="n">train</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">train_loader</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">,</span> <span class="n">epoch</span><span class="p">)</span> <span class="n">test</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">)</span> <span class="n">scheduler</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="k">if</span> <span class="n">args</span><span class="o">.</span><span class="n">save_model</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">state_dict</span><span class="p">(),</span> <span class="n">filename</span><span class="p">)</span> <span class="c1"># mnist()</span> </pre></div> </div> </div> </div> </p> </details> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Other out of distribution detector have been proposed. Here is a sample of methods:</p> <ul> <li>Genetic algorithms</li> <li>DFO</li> <li>Simulated annealing</li> </ul> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="Experiments">Experiments<a class="anchor-link" href="#Experiments"> </a></h2> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">train_ticket</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">train_loader</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">,</span> <span class="n">epoch</span><span class="p">):</span> <span class="n">model</span><span class="o">.</span><span class="n">train</span><span class="p">()</span> <span class="k">for</span> <span class="n">batch_idx</span><span class="p">,</span> <span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">train_loader</span><span class="p">):</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">target</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">zero_grad</span><span class="p">()</span> <span class="n">output</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="n">loss</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">nll_loss</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span> <span class="n">loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="k">if</span> <span class="n">batch_idx</span> <span class="o">%</span> <span class="n">args</span><span class="o">.</span><span class="n">log_interval</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Train Epoch: </span><span class="si">{}</span><span class="s1"> [</span><span class="si">{}</span><span class="s1">/</span><span class="si">{}</span><span class="s1"> (</span><span class="si">{:.0f}</span><span class="s1">%)]</span><span class="se">\t</span><span class="s1">Loss: </span><span class="si">{:.6f}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">epoch</span><span class="p">,</span> <span class="n">batch_idx</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">train_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">),</span> <span class="mf">100.</span> <span class="o">*</span> <span class="n">batch_idx</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">train_loader</span><span class="p">),</span> <span class="n">loss</span><span class="o">.</span><span class="n">item</span><span class="p">()))</span> <span class="k">def</span> <span class="nf">test_ticket</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">):</span> <span class="n">model</span><span class="o">.</span><span class="n">eval</span><span class="p">()</span> <span class="n">test_loss</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">correct</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">with</span> <span class="n">torch</span><span class="o">.</span><span class="n">no_grad</span><span class="p">():</span> <span class="k">for</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="ow">in</span> <span class="n">test_loader</span><span class="p">:</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">target</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">output</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="n">test_loss</span> <span class="o">+=</span> <span class="n">F</span><span class="o">.</span><span class="n">nll_loss</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;sum&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="c1"># sum up batch loss</span> <span class="n">pred</span> <span class="o">=</span> <span class="n">output</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdim</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="c1"># get the index of the max log-probability</span> <span class="n">correct</span> <span class="o">+=</span> <span class="n">pred</span><span class="o">.</span><span class="n">eq</span><span class="p">(</span><span class="n">target</span><span class="o">.</span><span class="n">view_as</span><span class="p">(</span><span class="n">pred</span><span class="p">))</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="n">test_loss</span> <span class="o">/=</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\n</span><span class="s1">Test set: Average loss: </span><span class="si">{:.4f}</span><span class="s1">, Accuracy: </span><span class="si">{}</span><span class="s1">/</span><span class="si">{}</span><span class="s1"> (</span><span class="si">{:.0f}</span><span class="s1">%)</span><span class="se">\n</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">test_loss</span><span class="p">,</span> <span class="n">correct</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">),</span> <span class="mf">100.</span> <span class="o">*</span> <span class="n">correct</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">)))</span> <span class="k">def</span> <span class="nf">ticket_finder</span><span class="p">():</span> <span class="n">filename</span> <span class="o">=</span><span class="s2">&quot;ticket_finder.pt&quot;</span> <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span> <span class="k">return</span> <span class="c1"># Training settings</span> <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="s1">&#39;PyTorch MNIST Example&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--batch-size&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">64</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;input batch size for training (default: 64)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--test-batch-size&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;input batch size for testing (default: 1000)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--epochs&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;number of epochs to train (default: 14)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--lr&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;LR&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;learning rate (default: 1.0)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--gamma&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">0.7</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;M&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Learning rate step gamma (default: 0.7)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--no-cuda&#39;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s1">&#39;store_true&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;disables CUDA training&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--seed&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;S&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;random seed (default: 1)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--log-interval&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;how many batches to wait before logging training status&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--save-model&#39;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s1">&#39;store_true&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;For Saving the current Model&#39;</span><span class="p">)</span> <span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span> <span class="n">use_cuda</span> <span class="o">=</span> <span class="ow">not</span> <span class="n">args</span><span class="o">.</span><span class="n">no_cuda</span> <span class="ow">and</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="n">torch</span><span class="o">.</span><span class="n">manual_seed</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">seed</span><span class="p">)</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s2">&quot;cuda&quot;</span> <span class="k">if</span> <span class="n">use_cuda</span> <span class="k">else</span> <span class="s2">&quot;cpu&quot;</span><span class="p">)</span> <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;num_workers&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;pin_memory&#39;</span><span class="p">:</span> <span class="kc">True</span><span class="p">}</span> <span class="k">if</span> <span class="n">use_cuda</span> <span class="k">else</span> <span class="p">{}</span> <span class="n">train_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">datasets</span><span class="o">.</span><span class="n">MNIST</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="n">test_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">datasets</span><span class="o">.</span><span class="n">MNIST</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">expanduser</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">),</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">test_batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="n">model</span> <span class="o">=</span> <span class="n">Net</span><span class="p">()</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">optimizer</span> <span class="o">=</span> <span class="n">TicketFinder</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">parameters</span><span class="p">())</span> <span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">epochs</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span> <span class="n">train_ticket</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">train_loader</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">,</span> <span class="n">epoch</span><span class="p">)</span> <span class="n">test_ticket</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">)</span> <span class="k">if</span> <span class="n">args</span><span class="o">.</span><span class="n">save_model</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">state_dict</span><span class="p">(),</span> <span class="n">filename</span><span class="p">)</span> </pre></div> </div> </div> </div> </div> </div> nicolas 2020-03-04T00:00:00+00:00 2020-03-04T00:00:00+00:00 https://nodata.dev/2020/03/04/running-gpu-enabled-docker/ <p>Sometimes if you want to contain dependencies you might want to use docker to containerize your projects. You can also use it for GPU In order to run docker images with GPU enabled, you are going to need:</p> <h1 id="install-docker">Install docker<a class="zola-anchor" href="#install-docker" aria-label="Anchor link for: install-docker"><i class="fas fa-link"></i></a> </h1> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>sudo apt-get install \</span></span> <span class="giallo-l"><span> apt-transport-https \</span></span> <span class="giallo-l"><span> ca-certificates \</span></span> <span class="giallo-l"><span> curl \</span></span> <span class="giallo-l"><span> gnupg-agent \</span></span> <span class="giallo-l"><span> software-properties-common</span></span> <span class="giallo-l"><span>curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -</span></span> <span class="giallo-l"><span>sudo add-apt-repository \</span></span> <span class="giallo-l"><span> &quot;deb [arch=amd64] https://download.docker.com/linux/ubuntu \</span></span> <span class="giallo-l"><span> $(lsb_release -cs) \</span></span> <span class="giallo-l"><span> stable&quot;</span></span> <span class="giallo-l"><span>sudo apt-get update</span></span> <span class="giallo-l"><span>sudo apt-get install docker-ce docker-ce-cli containerd.io</span></span></code></pre> <p><a rel="nofollow external" href="https://docs.docker.com/install/linux/docker-ce/ubuntu/">source</a></p> <h1 id="install-nvidia-container-toolkit">Install nvidia-container-toolkit<a class="zola-anchor" href="#install-nvidia-container-toolkit" aria-label="Anchor link for: install-nvidia-container-toolkit"><i class="fas fa-link"></i></a> </h1> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span># Add the package repositories</span></span> <span class="giallo-l"><span>distribution=$(. /etc/os-release;echo $ID$VERSION_ID)</span></span> <span class="giallo-l"><span>curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -</span></span> <span class="giallo-l"><span>curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>sudo apt-get update &amp;&amp; sudo apt-get install -y nvidia-container-toolkit</span></span> <span class="giallo-l"><span>sudo systemctl restart docker</span></span></code></pre> <p><a rel="nofollow external" href="https://github.com/NVIDIA/nvidia-docker">source</a></p> <h1 id="launch-the-docker-for-pytorch">Launch the docker for PyTorch<a class="zola-anchor" href="#launch-the-docker-for-pytorch" aria-label="Anchor link for: launch-the-docker-for-pytorch"><i class="fas fa-link"></i></a> </h1> <p>In order to use cuda you need a nvidia enabled image, that will make everything simpler. You could of course link your own cuda library via volume mounting but it's cumbersome (and I didn't check that it works)</p> <ol> <li>Create an account on <a rel="nofollow external" href="https://ngc.nvidia.com/">https://ngc.nvidia.com/</a></li> <li>Go to the create an API key page <a rel="nofollow external" href="https://ngc.nvidia.com/setup/api-key">https://ngc.nvidia.com/setup/api-key</a></li> <li>Generate the key and copy it</li> </ol> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>docker login nvcr.io</span></span> <span class="giallo-l"><span>Username: $oauthtoken</span></span> <span class="giallo-l"><span>Password: &lt;Your Key&gt;</span></span> <span class="giallo-l"><span>docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:20.02-py3 bash</span></span> <span class="giallo-l"><span>python -c &quot;import torch; print(torch.cuda.is_available())&quot;</span></span> <span class="giallo-l"><span># True</span></span></code></pre> <p>If you fail to login the <code>docker run</code> command will fail with <code>unauthenticated</code> error.</p> <p>Caveat: This is the only option for now, docker-compose <em>CANNOT</em> run the --gpu option. To check updates for docker compose, look at this <a rel="nofollow external" href="https://github.com/docker/compose/issues/6691">issue</a></p> <p>Bonus: Nvidia put up <em>a lot</em> of containers with various libraries enabled check it out in their <a rel="nofollow external" href="https://ngc.nvidia.com/catalog/">catalog</a></p> <h2 id="enjoy">Enjoy !<a class="zola-anchor" href="#enjoy" aria-label="Anchor link for: enjoy"><i class="fas fa-link"></i></a> </h2> 2020-02-26T00:00:00+00:00 2020-02-26T00:00:00+00:00 https://nodata.dev/2020/02/26/self-kl-models/ <!-- ################################################# ### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ### ################################################# # file to edit: _notebooks/2020-02-26-self-kl-models.ipynb --> <div class="container" id="notebook-container"> <div class="cell border-box-sizing code_cell rendered"> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <blockquote><p>TL;DR. By training two models in the same dataset order with same architecture, same loss, but different initialization, I was able to obtain a consistent out-of-distribution detector by measuring the kl-divergence between model outputs. This out-of-distribution measure used on text could lead to unsupervised text classification.</p> </blockquote> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="What's-the-problem-?">What's the problem ?<a class="anchor-link" href="#What's-the-problem-?"> </a></h2><p>ML models usually are not really capable of predicting how well the data you<br> feed them is close to what was in the dataset. It really matters in production models as they might make really stupid mistakes just because they are off<br> the training set.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Let's train a simple mnist model (straight out from pytorch tutorial <a href="https://github.com/pytorch/examples/tree/master/mnist">https://github.com/pytorch/examples/tree/master/mnist</a>)</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <details class="description"> <summary class="btn btn-sm" data-open="Hide Code" data-close="Show Code"></summary> <p><div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#collapse</span> <span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span> <span class="kn">import</span> <span class="nn">argparse</span> <span class="kn">import</span> <span class="nn">torch</span> <span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="nn">nn</span> <span class="kn">import</span> <span class="nn">torch.nn.functional</span> <span class="k">as</span> <span class="nn">F</span> <span class="kn">import</span> <span class="nn">torch.optim</span> <span class="k">as</span> <span class="nn">optim</span> <span class="kn">from</span> <span class="nn">torchvision</span> <span class="kn">import</span> <span class="n">datasets</span><span class="p">,</span> <span class="n">transforms</span> <span class="kn">from</span> <span class="nn">torch.optim.lr_scheduler</span> <span class="kn">import</span> <span class="n">StepLR</span> <span class="kn">import</span> <span class="nn">os</span> <span class="k">class</span> <span class="nc">Net</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span> <span class="k">def</span> <span class="fm">**init**</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="nb">super</span><span class="p">(</span><span class="n">Net</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">**init**</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">conv1</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Conv2d</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">conv2</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Conv2d</span><span class="p">(</span><span class="mi">32</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout1</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Dropout2d</span><span class="p">(</span><span class="mf">0.25</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout2</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Dropout2d</span><span class="p">(</span><span class="mf">0.5</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">fc1</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="mi">9216</span><span class="p">,</span> <span class="mi">128</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">fc2</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conv1</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conv2</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">max_pool2d</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout1</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">flatten</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">fc1</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout2</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">fc2</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">output</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">log_softmax</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="k">return</span> <span class="n">output</span> <span class="k">def</span> <span class="nf">train</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">train_loader</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">,</span> <span class="n">epoch</span><span class="p">):</span> <span class="n">model</span><span class="o">.</span><span class="n">train</span><span class="p">()</span> <span class="k">for</span> <span class="n">batch_idx</span><span class="p">,</span> <span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">train_loader</span><span class="p">):</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">target</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">zero_grad</span><span class="p">()</span> <span class="n">output</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="n">loss</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">nll_loss</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span> <span class="n">loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="k">if</span> <span class="n">batch_idx</span> <span class="o">%</span> <span class="n">args</span><span class="o">.</span><span class="n">log_interval</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Train Epoch: </span><span class="si">{}</span><span class="s1"> [</span><span class="si">{}</span><span class="s1">/</span><span class="si">{}</span><span class="s1"> (</span><span class="si">{:.0f}</span><span class="s1">%)]</span><span class="se">\t</span><span class="s1">Loss: </span><span class="si">{:.6f}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">epoch</span><span class="p">,</span> <span class="n">batch_idx</span> <span class="o">_</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">train_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">),</span> <span class="mf">100.</span> <span class="o">_</span> <span class="n">batch_idx</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">train_loader</span><span class="p">),</span> <span class="n">loss</span><span class="o">.</span><span class="n">item</span><span class="p">()))</span> <span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">):</span> <span class="n">model</span><span class="o">.</span><span class="n">eval</span><span class="p">()</span> <span class="n">test_loss</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">correct</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">with</span> <span class="n">torch</span><span class="o">.</span><span class="n">no_grad</span><span class="p">():</span> <span class="k">for</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="ow">in</span> <span class="n">test_loader</span><span class="p">:</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">target</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">output</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="n">test_loss</span> <span class="o">+=</span> <span class="n">F</span><span class="o">.</span><span class="n">nll_loss</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;sum&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="c1"># sum up batch loss</span> <span class="n">pred</span> <span class="o">=</span> <span class="n">output</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdim</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="c1"># get the index of the max log-probability</span> <span class="n">correct</span> <span class="o">+=</span> <span class="n">pred</span><span class="o">.</span><span class="n">eq</span><span class="p">(</span><span class="n">target</span><span class="o">.</span><span class="n">view_as</span><span class="p">(</span><span class="n">pred</span><span class="p">))</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="n">test_loss</span> <span class="o">/=</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\n</span><span class="s1">Test set: Average loss: </span><span class="si">{:.4f}</span><span class="s1">, Accuracy: </span><span class="si">{}</span><span class="s1">/</span><span class="si">{}</span><span class="s1"> (</span><span class="si">{:.0f}</span><span class="s1">%)</span><span class="se">\n</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">test_loss</span><span class="p">,</span> <span class="n">correct</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">),</span> <span class="mf">100.</span> <span class="o">*</span> <span class="n">correct</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">)))</span> <span class="k">def</span> <span class="nf">mnist</span><span class="p">():</span> <span class="n">filename</span> <span class="o">=</span> <span class="s2">&quot;mnist_cnn.pt&quot;</span> <span class="c1"># Training settings</span> <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span> <span class="k">return</span> <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="s1">&#39;PyTorch MNIST Example&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--batch-size&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">64</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;input batch size for training (default: 64)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--test-batch-size&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;input batch size for testing (default: 1000)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--epochs&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">14</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;number of epochs to train (default: 14)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--lr&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;LR&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;learning rate (default: 1.0)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--gamma&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">0.7</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;M&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Learning rate step gamma (default: 0.7)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--no-cuda&#39;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s1">&#39;store_true&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;disables CUDA training&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--seed&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;S&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;random seed (default: 1)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--log-interval&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;how many batches to wait before logging training status&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--save-model&#39;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s1">&#39;store_true&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;For Saving the current Model&#39;</span><span class="p">)</span> <span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span> <span class="n">use_cuda</span> <span class="o">=</span> <span class="ow">not</span> <span class="n">args</span><span class="o">.</span><span class="n">no_cuda</span> <span class="ow">and</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="n">torch</span><span class="o">.</span><span class="n">manual_seed</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">seed</span><span class="p">)</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s2">&quot;cuda&quot;</span> <span class="k">if</span> <span class="n">use_cuda</span> <span class="k">else</span> <span class="s2">&quot;cpu&quot;</span><span class="p">)</span> <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;num_workers&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;pin_memory&#39;</span><span class="p">:</span> <span class="kc">True</span><span class="p">}</span> <span class="k">if</span> <span class="n">use_cuda</span> <span class="k">else</span> <span class="p">{}</span> <span class="n">train_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">datasets</span><span class="o">.</span><span class="n">MNIST</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="n">test_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">datasets</span><span class="o">.</span><span class="n">MNIST</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">expanduser</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">),</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">test_batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="n">model</span> <span class="o">=</span> <span class="n">Net</span><span class="p">()</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">optimizer</span> <span class="o">=</span> <span class="n">optim</span><span class="o">.</span><span class="n">Adadelta</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">parameters</span><span class="p">(),</span> <span class="n">lr</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">lr</span><span class="p">)</span> <span class="n">scheduler</span> <span class="o">=</span> <span class="n">StepLR</span><span class="p">(</span><span class="n">optimizer</span><span class="p">,</span> <span class="n">step_size</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">gamma</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">gamma</span><span class="p">)</span> <span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">epochs</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span> <span class="n">train</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">train_loader</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">,</span> <span class="n">epoch</span><span class="p">)</span> <span class="n">test</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">)</span> <span class="n">scheduler</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="k">if</span> <span class="n">args</span><span class="o">.</span><span class="n">save_model</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">state_dict</span><span class="p">(),</span> <span class="n">filename</span><span class="p">)</span> <span class="c1"># mnist()</span> </pre></div> </div> </div> </div> </p> </details> </div> <div class="cell border-box-sizing code_cell rendered"> <details class="description"> <summary class="btn btn-sm" data-open="Hide Code" data-close="Show Code"></summary> <p><div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#collapse</span> <span class="kn">from</span> <span class="nn">torch.distributions</span> <span class="kn">import</span> <span class="n">Categorical</span> <span class="kn">from</span> <span class="nn">torch.nn.parameter</span> <span class="kn">import</span> <span class="n">Parameter</span> <span class="kn">from</span> <span class="nn">torchvision</span> <span class="kn">import</span> <span class="n">transforms</span> <span class="k">def</span> <span class="nf">attack_simple</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span> <span class="n">dummy_input</span> <span class="o">=</span> <span class="n">Parameter</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="n">requires_grad</span><span class="o">=</span><span class="kc">True</span><span class="p">))</span> <span class="n">lr</span> <span class="o">=</span> <span class="mi">1</span> <span class="n">optimizer</span> <span class="o">=</span> <span class="n">optim</span><span class="o">.</span><span class="n">Adadelta</span><span class="p">([</span><span class="n">dummy_input</span><span class="p">],</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="n">output</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">dummy_input</span><span class="p">)</span> <span class="n">entropy</span> <span class="o">=</span> <span class="n">Categorical</span><span class="p">(</span><span class="n">logits</span> <span class="o">=</span> <span class="n">output</span><span class="p">)</span><span class="o">.</span><span class="n">entropy</span><span class="p">()</span> <span class="c1"># print(f&#39;Entropy {entropy.item():.2f}&#39;)</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">zero_grad</span><span class="p">()</span> <span class="n">entropy</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="n">MAX</span> <span class="o">=</span> <span class="n">output</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">()</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="n">pil_img</span> <span class="o">=</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Resize</span><span class="p">((</span><span class="mi">240</span><span class="p">,</span> <span class="mi">240</span><span class="p">))(</span><span class="n">transforms</span><span class="o">.</span><span class="n">ToPILImage</span><span class="p">()(</span><span class="n">dummy_input</span><span class="p">[</span><span class="mi">0</span><span class="p">]))</span> <span class="k">return</span> <span class="p">(</span><span class="n">MAX</span><span class="o">.</span><span class="n">values</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">0.8</span><span class="p">,</span> <span class="n">MAX</span><span class="p">,</span> <span class="n">pil_img</span><span class="p">)</span> <span class="k">def</span> <span class="nf">check_attack</span><span class="p">():</span> <span class="n">mnist_model</span> <span class="o">=</span> <span class="n">Net</span><span class="p">()</span> <span class="n">mnist_model</span><span class="o">.</span><span class="n">load_state_dict</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="s1">&#39;mnist_cnn.pt&#39;</span><span class="p">))</span> <span class="n">success</span><span class="p">,</span> <span class="n">MAX</span><span class="p">,</span> <span class="n">pil_img</span> <span class="o">=</span> <span class="n">attack_simple</span><span class="p">(</span><span class="n">mnist_model</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;MNIST Model says : This is a </span><span class="si">{</span><span class="n">MAX</span><span class="o">.</span><span class="n">indices</span><span class="o">.</span><span class="n">item</span><span class="p">()</span><span class="si">}</span><span class="s2"> with probability </span><span class="si">{</span><span class="n">MAX</span><span class="o">.</span><span class="n">values</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">*</span> <span class="mi">100</span><span class="si">:</span><span class="s2">.2f</span><span class="si">}</span><span class="s2">%&quot;</span><span class="p">)</span> <span class="n">display</span><span class="p">(</span><span class="n">pil_img</span><span class="p">)</span> <span class="n">success_rate</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">attack_simple</span><span class="p">(</span><span class="n">mnist_model</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">))</span> <span class="o">/</span> <span class="mf">100.</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Success rate </span><span class="si">{</span><span class="n">success_rate</span> <span class="o">*</span> <span class="mi">100</span><span class="si">:</span><span class="s2"> .2f</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span> <span class="c1"># check_attack()</span> </pre></div> </div> </div> </div> </p> </details> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Then generate an random image for which the model is highly confident yet it's completely absurd. This new image is out of distribution yet the model does not know it. We want to avoid doing such mistakes in production.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="Other-approaches">Other approaches<a class="anchor-link" href="#Other-approaches"> </a></h2> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Other out of distribution detector have been proposed. Here is a sample of methods:</p> <ul> <li><a href="https://arxiv.org/pdf/1906.02845.pdf">Likelihood Ratios for Out-of-Distribution Detection</a>: Propose to learn 2 distinct models, one "raw", one with perturbation instilled into the dataset, and look at the log likelihood ratio of the two models, claim is that the difference between the two will reflect how "far" input is from the semantic part of the manifold of X. $p(x) = p(x_{background})p(x_{semantic})$, the perturbation needs to lie only on $x_{semantic}$.</li> <li><a href="https://arxiv.org/pdf/1910.04241.pdf">Out-of-distribution Detection in Classifiers via Generation</a>: Propose to use autoencoder (or GANs) to generate a low dimensional representation of the manifold of the dataset X, then perturb X on that representation. Those perturbated examples are trained to become a new "class" of the output of the classifier. </li> <li><a href="https://arxiv.org/pdf/1706.02690.pdf">Enhancing the reliability of Out-of-Distribution Image Detection in Neural Networks (Odin)</a>: This one uses temperature scaling regarding softmax to generate perturbated input, then look at the probability of the softmax if it passes a threshold. IMO, this paper is interesting as it supposes smoothness properties on In distribution data, and less smooth for out-of-distribution. It does require some examples of out-of-distribution for fitting 3 hyperparameters (temperature, threshold and magnitude of perturbation)</li> <li><p><a href="https://openreview.net/pdf?id=Hkxzx0NtDB">Your classifier is secretly an energy based model and you should treat it like one</a>: This one adds a new term in the loss to estimate p(x) basically. Multiple ood detectors are proposed, the most efficient being the second derivative of p(x), claiming again that density of p(x) will change more widly in ood space, leading to a good ood detector.</p> </li> <li><p><a href="https://arxiv.org/pdf/1810.01392.pdf">WAIC, but Why? Generative Ensembles for Robust Anomaly Detection</a>: This paper proposes to use an ensemble of models and look at WAIC criterion to detect OOD. It makes many comparison to VAE and GANs</p> </li> <li><p><a href="https://arxiv.org/pdf/1802.04865v1.pdf">Learning Confidence for Out-of-Distribution Detection in Neural Networks</a> : The core idea in this paper is to change the learning loss, to learn confidence as prior task to classification task, a model is allowed to see real label only when it claims it can solve the problem, outputting via another head directly a confidence score. Caveat is that the model might choose to give up and always claim confidence, and another trick is proposed to emphasize the in-distribution vs out-of-distribution by preprocessing inputs to move them towards region of higher confidence. In-distribution tends to move closer to 1 than out-of-distribution. So the direct confidence estimator seems to be <em>smoother</em> out-of-distribution than in-distribution, where peaks are more likely to be found.</p> </li> <li><p><a href="https://paperswithcode.com/task/out-of-distribution-detection">Papers with code</a>: More links on that hopefully</p> </li> </ul> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="Our-approach">Our approach<a class="anchor-link" href="#Our-approach"> </a></h2> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <blockquote><p><strong>Tl;dr : Make two similar models, with two different random initialization, then train them at the same time.&gt; The ood detector will simply be the a threshold classifier on the KL-divergence between the two outputs.</strong> The core argument for this approach is that the neural network captures the dataset manifold (which means it will produce "regular" outputs for in dataset items). For the range of possible values it has random values for a random initialization. If that is true, then we train the model, we shift it's output only on the dataset manifold, and not anywhere else. If that assumption is correct, then the 2 models have very low probability of concurring in their output outside of the manifold if they have been initialized differently.</p> </blockquote> <p>It's quite close to WAIC, <em>but</em> the two models need to be trained at the same time. The argument is that is should align gradients during the training phase, leading to more correlation for in-dataset prediction for the models. The argument for this supposes that the lottery ticket hypothesis is true, and adds that lottery ticket is unique (or at least that the class of lottery tickets is very thin, and they all highly correlate to each other). If this is true, then the gradients within the network that correspond to this lottery ticket winner in <em>both</em> networks should be the same (or highly correlated).</p> <p>In order to fix the threshold, we found that simply setting it to be 10x the average kl-divergence obtained on the train dataset worked pretty well. As kl divergence is measured in bits, 10x is a quite large margin. More work could be done to study more closely the behaviour of this self kl-divergence.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="Experiments">Experiments<a class="anchor-link" href="#Experiments"> </a></h2> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h3 id="Experiment-1">Experiment 1<a class="anchor-link" href="#Experiment-1"> </a></h3><p>MNIST attack like failure presented before.</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="k">class</span> <span class="nc">MultiNet</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">models</span><span class="p">):</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">models</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">ModuleList</span><span class="p">(</span><span class="n">models</span><span class="p">)</span> <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span> <span class="k">return</span> <span class="p">[</span><span class="n">model</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">model</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">models</span><span class="p">]</span> <span class="k">def</span> <span class="nf">train_multi</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">train_loader</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">,</span> <span class="n">epoch</span><span class="p">):</span> <span class="n">model</span><span class="o">.</span><span class="n">train</span><span class="p">()</span> <span class="k">for</span> <span class="n">batch_idx</span><span class="p">,</span> <span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">train_loader</span><span class="p">):</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">target</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">zero_grad</span><span class="p">()</span> <span class="n">outputs</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="n">loss</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">F</span><span class="o">.</span><span class="n">nll_loss</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span> <span class="k">for</span> <span class="n">output</span> <span class="ow">in</span> <span class="n">outputs</span><span class="p">)</span> <span class="n">loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="k">if</span> <span class="n">batch_idx</span> <span class="o">%</span> <span class="n">args</span><span class="o">.</span><span class="n">log_interval</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Train Epoch: </span><span class="si">{}</span><span class="s1"> [</span><span class="si">{}</span><span class="s1">/</span><span class="si">{}</span><span class="s1"> (</span><span class="si">{:.0f}</span><span class="s1">%)]</span><span class="se">\t</span><span class="s1">Loss: </span><span class="si">{:.6f}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">epoch</span><span class="p">,</span> <span class="n">batch_idx</span> <span class="o">_</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">train_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">),</span> <span class="mf">100.</span> <span class="o">_</span> <span class="n">batch_idx</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">train_loader</span><span class="p">),</span> <span class="n">loss</span><span class="o">.</span><span class="n">item</span><span class="p">()))</span> <span class="k">def</span> <span class="nf">test_multi</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">):</span> <span class="n">model</span><span class="o">.</span><span class="n">eval</span><span class="p">()</span> <span class="n">test_loss</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">correct</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">with</span> <span class="n">torch</span><span class="o">.</span><span class="n">no_grad</span><span class="p">():</span> <span class="k">for</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="ow">in</span> <span class="n">test_loader</span><span class="p">:</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">target</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">outputs</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="n">test_loss</span> <span class="o">+=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">F</span><span class="o">.</span><span class="n">nll_loss</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;sum&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="k">for</span> <span class="n">output</span> <span class="ow">in</span> <span class="n">outputs</span><span class="p">)</span> <span class="n">pred</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdim</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="c1"># get the index of the max log-probability</span> <span class="n">correct</span> <span class="o">+=</span> <span class="n">pred</span><span class="o">.</span><span class="n">eq</span><span class="p">(</span><span class="n">target</span><span class="o">.</span><span class="n">view_as</span><span class="p">(</span><span class="n">pred</span><span class="p">))</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="n">test_loss</span> <span class="o">/=</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\n</span><span class="s1">Test set: Average loss: </span><span class="si">{:.4f}</span><span class="s1">, Accuracy: </span><span class="si">{}</span><span class="s1">/</span><span class="si">{}</span><span class="s1"> (</span><span class="si">{:.0f}</span><span class="s1">%)</span><span class="se">\n</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">test_loss</span><span class="p">,</span> <span class="n">correct</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">),</span> <span class="mf">100.</span> <span class="o">*</span> <span class="n">correct</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">)))</span> <span class="k">def</span> <span class="nf">mnist_multi</span><span class="p">():</span> <span class="c1"># Training settings</span> <span class="n">filename</span> <span class="o">=</span> <span class="s2">&quot;mnist_multi_cnn.pt&quot;</span> <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span> <span class="k">return</span> <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="s1">&#39;PyTorch MNIST Example&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--batch-size&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">64</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;input batch size for training (default: 64)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--test-batch-size&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;input batch size for testing (default: 1000)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--epochs&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">14</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;number of epochs to train (default: 14)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--lr&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;LR&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;learning rate (default: 1.0)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--gamma&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">0.7</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;M&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Learning rate step gamma (default: 0.7)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--no-cuda&#39;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s1">&#39;store_true&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;disables CUDA training&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--seed&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;S&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;random seed (default: 1)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--log-interval&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;how many batches to wait before logging training status&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--save-model&#39;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s1">&#39;store_true&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;For Saving the current Model&#39;</span><span class="p">)</span> <span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span> <span class="n">use_cuda</span> <span class="o">=</span> <span class="ow">not</span> <span class="n">args</span><span class="o">.</span><span class="n">no_cuda</span> <span class="ow">and</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="n">torch</span><span class="o">.</span><span class="n">manual_seed</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">seed</span><span class="p">)</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s2">&quot;cuda&quot;</span> <span class="k">if</span> <span class="n">use_cuda</span> <span class="k">else</span> <span class="s2">&quot;cpu&quot;</span><span class="p">)</span> <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;num_workers&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;pin_memory&#39;</span><span class="p">:</span> <span class="kc">True</span><span class="p">}</span> <span class="k">if</span> <span class="n">use_cuda</span> <span class="k">else</span> <span class="p">{}</span> <span class="n">train_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">datasets</span><span class="o">.</span><span class="n">MNIST</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="n">test_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">datasets</span><span class="o">.</span><span class="n">MNIST</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">expanduser</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">),</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">test_batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="n">model1</span> <span class="o">=</span> <span class="n">Net</span><span class="p">()</span> <span class="n">model2</span> <span class="o">=</span> <span class="n">Net</span><span class="p">()</span> <span class="n">model</span> <span class="o">=</span> <span class="n">MultiNet</span><span class="p">(</span><span class="n">model1</span><span class="p">,</span> <span class="n">model2</span><span class="p">)</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">optimizer</span> <span class="o">=</span> <span class="n">optim</span><span class="o">.</span><span class="n">Adadelta</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">parameters</span><span class="p">(),</span> <span class="n">lr</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">lr</span><span class="p">)</span> <span class="n">scheduler</span> <span class="o">=</span> <span class="n">StepLR</span><span class="p">(</span><span class="n">optimizer</span><span class="p">,</span> <span class="n">step_size</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">gamma</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">gamma</span><span class="p">)</span> <span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">epochs</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span> <span class="n">train_multi</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">train_loader</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">,</span> <span class="n">epoch</span><span class="p">)</span> <span class="n">test_multi</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">)</span> <span class="n">scheduler</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="k">if</span> <span class="n">args</span><span class="o">.</span><span class="n">save_model</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">state_dict</span><span class="p">(),</span> <span class="n">filename</span><span class="p">)</span> <span class="c1"># mnist_multi()</span> </pre></div> </div> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">torchvision</span> <span class="kn">import</span> <span class="n">datasets</span> <span class="k">def</span> <span class="nf">kl</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">):</span> <span class="n">model</span><span class="o">.</span><span class="n">eval</span><span class="p">()</span> <span class="n">test_loss</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">with</span> <span class="n">torch</span><span class="o">.</span><span class="n">no_grad</span><span class="p">():</span> <span class="k">for</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="ow">in</span> <span class="n">test_loader</span><span class="p">:</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">target</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">outputs</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="n">loss</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">outputs</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">):</span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">outputs</span><span class="p">)):</span> <span class="n">n</span> <span class="o">+=</span> <span class="mi">1</span> <span class="n">loss</span> <span class="o">+=</span> <span class="mi">1</span><span class="o">/</span><span class="mi">2</span> <span class="o">*</span> <span class="p">(</span><span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="n">j</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;sum&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">+</span> <span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="n">j</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;sum&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">item</span><span class="p">())</span> <span class="n">loss</span> <span class="o">/=</span> <span class="n">n</span> <span class="n">test_loss</span> <span class="o">+=</span> <span class="n">loss</span> <span class="n">test_loss</span> <span class="o">/=</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\n</span><span class="s1">Test set: Average loss: </span><span class="si">{:.4f}</span><span class="s1">, len </span><span class="si">{}</span><span class="s1"> </span><span class="se">\n</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">test_loss</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">test_loader</span><span class="o">.</span><span class="n">dataset</span><span class="p">)))</span> <span class="k">return</span> <span class="n">test_loss</span> <span class="k">def</span> <span class="nf">get_reference_kl</span><span class="p">():</span> <span class="n">multi_model</span> <span class="o">=</span> <span class="n">MultiNet</span><span class="p">(</span><span class="n">Net</span><span class="p">(),</span> <span class="n">Net</span><span class="p">())</span> <span class="n">multi_model</span><span class="o">.</span><span class="n">load_state_dict</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="s1">&#39;mnist_multi_cnn.pt&#39;</span><span class="p">))</span> <span class="n">test_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">datasets</span><span class="o">.</span><span class="n">MNIST</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">expanduser</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">),</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="k">return</span> <span class="n">kl</span><span class="p">(</span><span class="n">multi_model</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="s1">&#39;cpu&#39;</span><span class="p">,</span> <span class="n">test_loader</span><span class="o">=</span><span class="n">test_loader</span><span class="p">)</span> <span class="c1"># ref_kl_loss = get_reference_kl()</span> </pre></div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Now we have 2 models capable of detecting digits, we have instantly 3 checks for checking if the output of our model is valid. The 2 models need to be concording (they need to outputs the same digit as an output), they need to have similar kl-divergence, we actually have a reference for the test set, so we know what kind of divergence we should look for, anything 10x more is definitely ood (we could look at the test set distribution for more fine grain estimation). Because kl divergence is asymetric we have 2 values (it's harder for spiked distribution to have another distribution be close in the kl sense, so taking the max of kl-divergence should be used for out-of-distribution.</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <details class="description"> <summary class="btn btn-sm" data-open="Hide Code" data-close="Show Code"></summary> <p><div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#collapse</span> <span class="kn">from</span> <span class="nn">torch.distributions</span> <span class="kn">import</span> <span class="n">Categorical</span> <span class="kn">from</span> <span class="nn">torch.nn.parameter</span> <span class="kn">import</span> <span class="n">Parameter</span> <span class="kn">from</span> <span class="nn">torchvision</span> <span class="kn">import</span> <span class="n">transforms</span> <span class="k">def</span> <span class="nf">attack</span><span class="p">(</span><span class="n">loss_fn</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">n</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span> <span class="n">multi_model</span> <span class="o">=</span> <span class="n">MultiNet</span><span class="p">(</span><span class="n">Net</span><span class="p">(),</span> <span class="n">Net</span><span class="p">())</span> <span class="n">multi_model</span><span class="o">.</span><span class="n">load_state_dict</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="s1">&#39;mnist_multi_cnn.pt&#39;</span><span class="p">))</span> <span class="n">dummy_input</span> <span class="o">=</span> <span class="n">Parameter</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="n">requires_grad</span><span class="o">=</span><span class="kc">True</span><span class="p">))</span> <span class="n">optimizer</span> <span class="o">=</span> <span class="n">optim</span><span class="o">.</span><span class="n">Adadelta</span><span class="p">([</span><span class="n">dummy_input</span><span class="p">],</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span> <span class="n">outputs</span> <span class="o">=</span> <span class="n">multi_model</span><span class="p">(</span><span class="n">dummy_input</span><span class="p">)</span> <span class="n">loss</span> <span class="o">=</span> <span class="n">loss_fn</span><span class="p">(</span><span class="n">outputs</span><span class="p">)</span> <span class="c1"># print(f&#39;Entropy {entropy.item():.2f}&#39;)</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">zero_grad</span><span class="p">()</span> <span class="n">loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="n">MAX1</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">()</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="n">MAX2</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">()</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="n">kl_loss</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;batchmean&#39;</span><span class="p">)</span> <span class="n">kl_loss2</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;batchmean&#39;</span><span class="p">)</span> <span class="k">if</span> <span class="p">(</span><span class="n">kl_loss</span> <span class="o">/</span> <span class="n">ref_kl_loss</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">10</span> <span class="ow">or</span> <span class="n">kl_loss2</span> <span class="o">/</span> <span class="n">ref_kl_loss</span> <span class="o">&gt;</span> <span class="mi">10</span> <span class="ow">or</span> <span class="n">MAX1</span><span class="o">.</span><span class="n">indices</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">!=</span> <span class="n">MAX2</span><span class="o">.</span><span class="n">indices</span><span class="o">.</span><span class="n">item</span><span class="p">():</span> <span class="n">success</span> <span class="o">=</span> <span class="kc">False</span> <span class="k">else</span><span class="p">:</span> <span class="n">success</span> <span class="o">=</span> <span class="n">MAX1</span><span class="o">.</span><span class="n">values</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">0.8</span> <span class="ow">and</span> <span class="n">MAX2</span><span class="o">.</span><span class="n">values</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">0.8</span> <span class="k">if</span> <span class="n">verbose</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;MNIST Model says : This is a </span><span class="si">{</span><span class="n">MAX1</span><span class="o">.</span><span class="n">indices</span><span class="o">.</span><span class="n">item</span><span class="p">()</span><span class="si">}</span><span class="s2"> with probability </span><span class="si">{</span><span class="n">MAX1</span><span class="o">.</span><span class="n">values</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">*</span> <span class="mi">100</span><span class="si">:</span><span class="s2">.2f</span><span class="si">}</span><span class="s2">%&quot;</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;MNIST Model 2 says : This is a </span><span class="si">{</span><span class="n">MAX2</span><span class="o">.</span><span class="n">indices</span><span class="o">.</span><span class="n">item</span><span class="p">()</span><span class="si">}</span><span class="s2"> with probability </span><span class="si">{</span><span class="n">MAX2</span><span class="o">.</span><span class="n">values</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">*</span> <span class="mi">100</span><span class="si">:</span><span class="s2">.2f</span><span class="si">}</span><span class="s2">%&quot;</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;KL-divergence is </span><span class="si">{</span><span class="n">kl_loss</span> <span class="o">/</span> <span class="n">ref_kl_loss</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">kl_loss2</span> <span class="o">/</span> <span class="n">ref_kl_loss</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span> <span class="k">if</span> <span class="n">success</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;ATTACK SUCCEEDED&quot;</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;ATTACK FAILED&quot;</span><span class="p">)</span> <span class="n">pil_img</span> <span class="o">=</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Resize</span><span class="p">((</span><span class="mi">240</span><span class="p">,</span> <span class="mi">240</span><span class="p">))(</span><span class="n">transforms</span><span class="o">.</span><span class="n">ToPILImage</span><span class="p">()(</span><span class="n">dummy_input</span><span class="p">[</span><span class="mi">0</span><span class="p">]))</span> <span class="n">display</span><span class="p">(</span><span class="n">pil_img</span><span class="p">)</span> <span class="k">return</span> <span class="n">success</span> </pre></div> </div> </div> </div> </p> </details> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Now if we simply attack the first model like we did earlier, we can see that we can trick it as easily as before. <em>BUT</em> the second model, does not get attacked which is to be expected.</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">loss</span><span class="p">(</span><span class="n">outputs</span><span class="p">):</span> <span class="n">entropy</span> <span class="o">=</span> <span class="n">Categorical</span><span class="p">(</span><span class="n">logits</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="o">.</span><span class="n">entropy</span><span class="p">()</span> <span class="n">loss</span> <span class="o">=</span> <span class="n">entropy</span> <span class="k">return</span> <span class="n">loss</span> <span class="n">\_</span> <span class="o">=</span> <span class="n">attack</span><span class="p">(</span><span class="n">loss</span><span class="p">)</span> </pre></div> </div> </div> </div> <div class="output_wrapper"> <div class="output"> <div class="output_area"> <div class="output_subarea output_stream output_stdout output_text"> <pre>MNIST Model says : This is a 3 with probability 99.32% MNIST Model 2 says : This is a 3 with probability 33.50% KL-divergence is 587.7392578125 152.96902465820312 ATTACK FAILED </pre> </div> </div> <div class="output_area"> <div class="output_png output_subarea "> <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAPAAAADwCAAAAAAbPrZOAABr+klEQVR4nOz965YkuZE0COoNgJm5e2Rmkezu2efdR93eM9+wqjIzwt3MAOhlfwAWWWTPzAPsrvOQh3WLCrjioioqIor/TwDAnEsen5JzyTn9MT96v93v9/vt/ucfv//++++///4qJZeSS0k555RzplrPWs9at3+Mz28f7x/vHx/vHx+v5+v5fD1fX//2229/++1vv31Z1mVdlnXR8zyP8zz333//4/fff//j9/z3v//j73//+9//vqzrsi7ruv/3f//3f/+//vu///vx7du3r9++fb0nGZ/3P/744/c//vjjebvf77f7/S4BAQERHx8f7x8fHx8VAgAC4D//67/+t//63/7rv76omZqaBQAQ/P/Y5/+/4P9v/0gAAMRfPz4/448AABCRiFlSyj2nlFISEWYmovFXWMyTCBMhjr9/fK5/CyIizb8Y4R4R86cSETOPn0X06x9CJGaR+W/6/GvzB41/AOcPG2f2148EB4AAiHAz1d7VzWz8S0EUAJCI2czclCAiXI+z1tpaN3dAYsnLrWmQlCPllHLKSURYWASRpfSuff3t69t9LULE4zsQpvFbIbHkUtY1CYNpdWvqgZxKWdbbWXvPj/v9tq2l5CRMCAGc19uXs9n9y9uXt/ttLYnHvzCXZb3V1mlZEodWZEBAQAAPJMlFyd3DwUPr8fyZGY4AGN/KrwWzKZsbIUKEK5/nWVvr3S0CiVNabhqU8nqmJJJEEs0Pcspu5r58+fp2WzPPCMzIIOL4AWVZVxYC76GuauO3W7ezNtX09rjf1mUZCyYI5LzevzSF7XF/3B+3dRFmYWZOZVm31pUlZQqrzohISIBjwepk6maBbu14ZkHfcewf/uuCmc3MCDDcjfk4z1pb62FXhM1JynpvwszCwjA3b6TxKkB5PB73Ncu1XmYZAQYglpSXdQVC8DAMdwNkhGVZb62b5bfHbdvWpaTEnxG+NwNeb7fbdrutCzMzMXMuy9pUnYmIQL0TMTIyoQeypAKkXRHcQ+ueGa0/U5KUJCUAAOkAMM+RGQW4mTKNLd07WAAQp7QApbLdj05MRMQUEeHh4ERESIRlu23bUgTx15IJ8S9beglwcI+AAAhkprJsrat7Glt6KTK2NADn5a7AuazrumzrkpmZiIlTKWtXC46ICI0gZhJiJgMgyYHUCMERQush6L1upSylFAe6IkwkZmZm5GFkRHgcZ22tdTQPJJaMJGVttSkhIiGiu5qZuUliERHJZVmW8q9bmokQfm1pM1Mz0/ETCLmsvatFyNvjtq3rkue3FMB5teC05FJKKUtOPI9Qzks3DxTtat26ogiLiLMHsgAyE0YYQVg70Hvd13Xb1s2B/7JgZRYVV3RCQsTzPGttXdEDkCQlTsVUVR0QAABBtXftXW2maDmlnFJKcl28Ikz/tqWbBnjvjZhZiBmWpmoOMCK8LIVpbmnJK3Be75LGzxUiQkIiLd3cgbierYLWikkkmUt4IAmRMEK4IrhW9H6+ynq/1x7Amf+6YFE2my8KwHGOCNMVYQAfmxgCHCKit9pa5aZlXZZlWVYhIWJiJGIaQb8uLRyX64qg4FpPTgkDOWFZzTwQ5e1xu21rKYREiBDIGTivtbGQsDAzEhIQkql6ALG8MHpo3SGn5A4BEcgYHAxuSoShEO2UJMtbVUfKy1+eJWUVM4NACISI6wyzBSCxJB7fL4KHu3tYO8+jMgKWbdu27bbxiD04/uWWxutZSqWsaxiBaz3EkQRZpKuP3/5+v2/buhQERCAI4MzZTA0JiOYJQEDEVCwASQS9U2g9QrOFA4znAALZTTsjhFpHQqJS1YHTogEAYgBAZtoZCQQIEYhAJKWcc5ackggTXq8QuCEEerir9tZqT+pAKS9k7mbmeuz7vh9nbeqAJLkkBu/n/gFnrV0dmdBNEYzP1j2IxyUqIgzuEBHg435QH2ElICQkxHEXJwtCU+2tJo6ZlxABASFiinBX1QaIgBA+7ifVHlem5a5EGKFJWEiE2dzcw4Fva0lCGA7hSIhgaqZmWo/x0SBORT1Ce2+99XYex3kex7nX7ih5tcJeXz8SfqirGiYCQm/9JKi11totPrOo8Z2Zm3ZTVVViJCZGHvkMkXUPlEDU8THPaZxzIkImoj4SMLexWxDzkoUgfETYAYBMEcFNLWcmKSm5W0Q40LYuWQjBYaQzofN3OY9jP479UJRUmhq41vM8zvNoZ6211rOe3ZCzwcLRXhn1hohIkHJ4WPdw71177xYAiIBIZK69a9fee+9dWx95pxALC5OwuBmQILOqjnciiSRJknhknNIiRlKJQIiIlJYsjGDdf0XYANxMHThI8lJmKg14W0sWwgAb6ZmralPtehz7sR/7oZLKsqmDa91fr9fr1VtrvbXWqzpyAV7Y2wv7saWcUkkpeVPrvffu5uY2vsxxXVpvtdVWe229tdbG7ScsSSSxJIGIIOSQuV5zYZGZd4oISx5JtBmMQ0CylEQYc8EOAGgQ4MoclAIlr6vP/ABv65KFIQIcHDy899567/049v3Y991SWbemHtHr/vH+/vGhXbtqVzVzEKSc2Svo8VzW27pxSlsP9XacRx1J/tg8iEgUrvU8zqNeH0mSUkqSsuQkOaVxlAnd1Ma7LrP8SDLugh7uZqaK454lWbIwhutnhMHBnYgIkgVJWW4eHgABcJ3hiHE9W2+9tt76sb+Ofd93K+tW24zw8+f3Hz9czVzNIiCQOQVTND2Jy1szKphWVrS6v577zHFprhcRTNux76+jHud51uNMOeULXCmp5zweKWGYAQ4lIiYizpJTTjlp+KiScNxlJEsWwrBO14IjHB0BERcLlLzePBwBAGNb59cT5uZm1mqrrdV2vPZ931+7Ldv97OoQWvePn3/+8We4e3g4EhETE7l7d3dPp2G6QdrwRGvH+89nkiSSAD8ryvBez/3juc8r8Ugl5wkqlZaLWU7IKDnhTPRiBBKJ0kSo1M1V+1gwERGXnBjCFD+3tMN4RLlpEOf1FhEIAOjrVrIQRIzto9rOdtZ2tv31Ol77a7fbfa/NPFzr/vzx5z9/n4hSiKTEkpL0aq22WsUob4ppjRdY2z++f+RScgESxFnLgmk79+f7c5+fvJRSylrKUpa1mFsgB0la2MZzEx0REAEp55EzzzdIG/IIPicR/rcIR0BEhHRz4JQXVXP3CC8lC0OYmaqaaq9nPc961v312vf99fLHftYZ4eP18ePPfyJeKQIySinlsNr31+vFvGxvPWRRBm/H6/3Hupojp1G6IwC49Xq8Xh8fr9frtb9er7IsS1nOZWmLWjd3JAmglFPXbmbmeoEMOZelLMtivffeWm/IY4cxMzFBmAOATHxhRIWjnx8M/fp+d89nziWVbD6gP2211dpqO47aWlf7fNcdJS+3t1PHho5R4i/bugp4b40xtJ2vjx9bPp/VqNy+ShmfLASurR7w2o/zrLV2B84WKCP9wfCZGgW4m2pv6XWMd57n90vMaFVPttfrqIZpmUkpy1w04bVgHEtG4OiHgB7v5zGQVE8pS5Ys7maubtpbb623fh5nbV0tRoKl5sB5uT2az4wAAjmV9Xa7sWurQuhWz/19XaQ9T6Nyi5JySimnuWAm2/f9OM/augGnICnCIsIEpkQI4O5m1nur6fpafFSnRBGhHSK0nq05poWTiCQRZsYRZAAQvEArxLFg6Oez1PEa1uAknDixu7u5j9uga9d61rN1NZgRtkDOy60Z1NZaBddATnnd7g/QXg9hDG3H66MIej0N8002EWEWkbFgBH29juOstXUHSijZRtaI4aaEEeN8tl5r6tq7I2egsXO5t5GwqKqpoSxpfmSe5V8RBqRxhXPoqedTpI8cqMeorZk93MPCXcdhtlZba10NfXzrCsh57Rp0nAeDawSSlGW7v0WvexaC0Ha+ihCgmVOR1ee7QULgiuDtNSJcNYBJJoYIAOGGCBGqpr23Vs7k4W7ASDzRnyPUzuPYx6HGNKqBnPIvOOpzSyPy+AqidwQEmKW6BY4yFEdtGOHmZm7mvfXWuxqN21INUfJiQfJ6EVjHmFv68cXrsWRhdGtHTgQmjEwyAD4AQGCE0PBOr30fEUYaWdIEUM0QIMy4a++91ZwTIiCgAMtsSkA/tT4/3rsIC3PiXMZz9ln8wF8iPLIVNzVVU5vXTgTgyINiJFsDxQ33uLKpa0srA6cFUHIi8F4JAjnlZbu/6bGPx1zbKRTWSykll1zSvN8CEDxcCV77cRxnbSpEIiJi80Jwg3A2Eu2tpZxy4hnZlGYypgfa+fzxZy9lyUWkLEtZSlmWRHMD/+UM00DmxrNTzzq/+V8v1pUEjj+IgNGtUYNxaZkRcgaWso6al2aEt/tbfz1LToSh7USwXrd7CObbfTF1NXWNAA+A8Ne8ph2FU8m59NYahoWBuyEjj1xzVq8InMpIw1I6E9r5+vHPfrs5JEyzkbWseSSj+C9bmpglcUQ/nq/Xc/9V/rpbuI/8PhBjfhPo5m5uhtelxciZUl57eG+7/GVLt4+Zrlkj9173t+CVyu3rrXftvSmZuZu72Wsfl1YIcCrrutaTMQzDARwRgTixJOGUljVQcO7bXHL6SGj1+eP31hxlwbSs27puy/ZrwX9BLQEppZLEemUMV6SBNECABwQgBn5GGSBwZioB4dpb3V/PxcNRKNt57lmYYGAlkkZrcvQQiAAiAokklWUVViJkVIgw095r0wCSDHk2Mq7s0BACHMECHCACQLpIn0AhERGq9jY/vaupRQBxyktBBMKRbIAcAEDAwGldl84iknJefF5ReGUq4/zC3N8OMdKQCLB+7h9LlpWQiIkkpwE9/urgIEkqZdWBfi3r43FbSyKIQCROCBiOYb21HpSC8hqSEkWHrq2pIyeEGPD32IspJUbv4P0ss80rf/58NUNZODFYOwQ8ULIFIAb4vO7HglkycF5uNxORVEpeZplvM9XJCWZF4OHjfVLtHSAcrNf9IzNuKaWUSTinCS7HWLIDMqe8rLat67Kt63p7u61FGByAOAAJ3XpYb6cHJco2MBDvCq5mAQx0PRNIzCmlnBi9u1Ya9VTK8sf76zRIC2cKawcaIOcycNbPNtlccF6A0nJ7c0kpl1zW2mptYU6U13Xd1nUWS6NicnVvnRDCELyfexaM27IuwWmsl6/G2YARSFJe1ti2bR2Q318iLEQM2jFM64lIlAEhxs1vChERyMTh7ugeSCSSck4I7h0AZGYX8vOKcGawBt6QU1nNYb6pHp8LltWC83p7C0kpl7Ks+3EQWA+ntNwfj8fd9bqWRxFhzADuRmb93Bldz7sGZZQyjt/nen22QMoKo21yuy3rsmQh8ABkJPYQgrDe6sgkRby2WntrFfFCS83cAAKRWVIuaXz1ZjSbXfx67qeCLMIM1rwLp7I2DUSAcHP3+Fzw3YDTcnvDlHNZlvVVEoc3cKK83r98+/rVuup4ekcSpkQRZoRgve4Y2poGJQPOoyGG+NmIvSKMt9vjdr/d7yXnnIXGZUgBEJUwrLdzIUqllOKvPXr04zWwLBI0IgUIHBdhyqk319Z6wwntcD1qM5RFEcCsI0hZa58RdrdBeZADAFIzoLze3iiVsizLuiYG7YIOlNbH17/9/W86kZ02EauO6G5KCN5PdK1Hd06rwRVhmm92hAOOlhDd7vf72/3+SElYhMAn7o8mBG7aKidKy7ZtztEP78dHyikIObEqAgQ5EY0IR/d+nucRsyQi66oGsoyD5+55vZ1NHcjB3dT0c8HaZoQll7Ks67ERWDsZHSmt9y9/+4//0Hnh11ZrbbU2ALPOTGAdQtvxUspLM+ScRwsQB9UkPAJHq4Vv98fj7fH2xjw6nIETZrNRoLczzzNk0U+OfjyzBSXgzIQQ4Qjj0so5dfR+vl4vH8k+8WjRJbHezFrXXm5H7RaAA5yeuPQBANYMOC+3t5SXZT3X8whrxy7oSHm5f/3bf/w/eh0LPet5njWJhKs2IQRrblVYLK339ivCCJ+HGMaztMjt/nj78vb2hcbzBk6AxExs4wzX2oPScnv7Zno+ydvxsQCKI6dEABFGc0vnkip6r/vHu05cjJhFSIQcrVur57k8jtENgAg37S0AQP4BAOnLJlGf33NqvfXWDJA4pVwWEvB+Pt/XPuJaazvn/2tdLQDpqmm/3DJ6ff7AP3+8v47a3c20t1rPZsB5VbD7VhisHUSAgECI4eGGeLZRXbYlMYb1qrV1NXdwM21CmFwdKJGkzOhase3HaIA5I5DwgGlZhJ1GywGzoFs7X2zdTB0oAED+EwDk65a8foinuf8DaDydKGh1/yjU28Du6nWGz9bUHZAlL0tZyvK4F7b6pP779/ePvXYL195bPY5mwNlBbC2JQ08crV5CivlWnF2DJC+2ZMbQuutZWzcHANfOFJEgAJhSSBL0Dk6vUVepExBLHvfCWPDs2C5CoO14YYzsGBEA5L8AgO6bxPlupwDOb+d6SQSs7e/kn2f4uqXr2bsFIKdlna/rwnaSHX9+//ncz25gqq3W81ADzsjZkyQOBeckEoJIHuYR7rUNtDSWLODaDj1qU/XAcOsE7onpavUzhnqH134cZ2sdHJBTLvlzwZMsQ0uisHY8R1Y8GUvyX6NYFq9+vqfR9ROCec+4oNedoutEdtqoH1Vbb2oOBCmvt/v9fi8pkVev/OPHz4+9dkPT3ls9j3DgLEUBETFUa8rZARnJJyLWmgZKdiwjwtDP2tU8ANw6hFlKkoiSJAwANwAbEe4dExCnsiy/Ijw/JRFYO1JMgHpE+D9hIPFxHhBSci6lQAIa6a8JWNtDz9lwaH0+995H7x4pLevt8fb2JQGA2Qnx8fPjuZ/d0Ux7Pc+DEJkRAdzcrbsl8yAOJAxT1d61WRBnFJkpcptbGsMVwrmnpTBwKgV0fEf92I+zttbZAyXlZfv3BfOSKLQd5OOAT8TjvwDA51tT07pumwHLjLCqoNXo53O2t7r6TKjNVM0BKeXt9vb16zeaf0d/Pl+vvXYj095aPY80fhHG1npTbS0HkCRAgnDrrTU1HbUeMqKr93bUNiMM4EqcgBJQWjZvzb331o6x4j5qorKsv7b0rwWDVQZLKXsioM8I2/PDj/Pj+SGPezPgHDhYCtoJLDoR6ejpqcb1pIx+KnJa1vvj62+/wX74eR77fuzHcZzdYFxa5xkonEvJeByHgZ5HR+LkgYTh2tpZLSJIOI2nWyHqWVtXDwwLBwJMlAw4lU3Bu/d6nOe4pDs6EKe8/iXCs4OTEoc1tF7KaLHSFWEVO6N+/PGHfK0GnNeRDRY1vhqbOjJL1YFDTcxnPEvr7fHlt787++n14+d7O2urrTtelxYKcF63lQWshtaXMie1QEJw7fU8nJBQkNDM1M3sPMYZRg+HgICUigPlZeuu6Fr34zzr2Wrv5DAA8P+xYBYKBe9NHYgdiABANgDQVxYM69Xa2EhIozA2MDMINeszibYJew8SICLmbdvWbV1XHS0KbV09AAcxAyHcxuNVVq5CYP3cTVJW84CBqdcziJmRkSMcwd3GT8jFfFJLLihhoSZM46cCkVjKeXIhRSRxEvYLmwCCUFcaBzTms/QBAFoNy61Dkre3L1/ebmsxVRncQYbwCG+11UroPrNfnI1byV+/bIW8vqw55c0ot9paq63x2+Nxv62DbkYQFtp7q7WeUGtpfcBzvffWPi/WiHBMkoVEcl7Ww0dRDfJ4PB63JRGnVNQAeOm9aW8932/3Rcg7IAUgiQ/KZzhAOCKipzQ4lAgA8g4AfhrlG6RV7vf7435fi/aUhJl5LA/wPE4hCLXZukLOJS+55OV+u2W0+ozqmDfKWz1rq7VWejzug1+XhRHcXHtrrdYKpdaBmM96xHH06gf0xIR5pAHnGRNS4m27bduS2CSrBaLoqFNVlmVZEnpH4kBkcZl4C9jAIm3sp/gVYZ9924fc1m3btjX33kVGUi7ExLyPdghfnTKQvIy9XJaSySpAd8qUVz3P86znedL9/rjftm0pSQghPPq4tk+oA3hKc8XNxrcKKEmERCQkl2WttU6mBfKFuLKk4oCU3MzdzElSkoTeiD2AiIkv6GySSr11VY8Bw8oHAIQ75bS6y1LKUpYl9zbYOwP1lSRFCF0b00zEgPN6e9zvj42ZiLwqQFDOEHAcx3ke54H37X67beuaszCFmw8QvZ6w1NZGhEfZqQMBBswBjFIKpNJaa71NJgTR6O8zsSQHYikTOPd5mViIeCCyxKCHwfjLEWFl5OW/tjQycWFmksmIkd6aCDNHKmXoIEanhGm03RA4r/e3L1++3D3c3dTpuiX3/TiOYy9w27bbtq1LEmGEMOu9tVrribW29pc93fRCv1ekTFJWnh2sPjofyDihSXIJIJKsn10YG30vS+MeY5ik1nCPgTG3kbX92tK4lFTKUpaxi5ko5yvCeVnXsq4CbvVMjJNEC5zX29u33357tFZb09Z4KZSXsuTRaC0ZtnVb121dhJlx3MhjS9NngHvvrbXWr04HcA6UvCZ1VTPTUfohY/jgAnIAsqgaznOvrbfe3bXncbfHL+74hOGuJOYzwgyJ8u1xvwtOnH3uaIK0rLdtu23o2o8jEV29F87L/cu3v//j7bW/QK3uTInydr9tr9drXUpOsS7ruqzrwkgEEDZu5FpPqrXNJc9ry200kTiro5Qtu7uFhc3inmE2YZUBWcwtaHYN+3EcR3Rv2TwQSWJCiLONa2pr/7dLS9JG5f712zfxsLAwrxf2OEuDB1irr5KYLnIp5/X+9u3v//H150/Q0+sz5YXy9vbl7bmtr5JT8qUsa1mWMnQVPpDyWmvlwWv8S4hn+yLyokFS1tXHpeNXaxAmAOGBHB7hMLlKUj+eHHra2LZEDKMzGuFu2lXVW1ez6wzfAICLoPfzxeSDOWWv1+vUoAQpD9g3MSNG+ABUSOi3r18et20pKedcSlma5MQIYeoD4sg+rqqOoyBHGGSi3hodpZTEZMfRHaV4mrHwQYkHiACfe9hIiThaa7W31h0CAJCBJzWPEALCfeYoiGMztXqamwcyMIbW45mQAUB+AwBaC+oB7YXXgmuttRoI5TRIovNMqDEPoP/r374+bksiJE556R68ZkZrJw1chRgJ3Dq4XhTygaRpb3TuSTCidVWnzFl1tC3ttpbECDaYBe7zgUaKK5mPKXHACEdDAB1cyClLgYAB59TzcAhAJhEOPV/ola4FozDp0V4ME2v3sfcxDSSB5q1nZgayrOu6rV++fnu7LYmAJC/qgLgWBm8UdSxYnDAs3GiS9yl8XlyQhQncKwQEE8RFkxgLpnCdRaABBiDERcI0g3lZDUzPItR08DPnigHGgo8dhugGE4NWtLZ/Lhg8XLvH1RE0n69tSleErzNhKHl73B+Px+PtsY0Fp7wGEuQkYB2s9z4WjOBg1lFEWGLcIqq9dziI0FXPUTaK+AS9Z4TDsA+cwca+/sv/4tXMj3AA8Jicy0t3BBBh2tt5HigixDwibG1PeC3YW61Wa616RXiweZOUlBNPPZKHqxrwsj2+fv16v223rSRG5LQEkDgTozWtQyOEjKPSg5CUksfQj1jX1owJwrTXsixEZSlwFZ/LVrIgWOh8h93cwtxiPg6jdQ84UmwP8j4ZaubuPk+/9lqPXQKIKaWE0K2N9vBYsD2h6fF8PusV4XVdlgVzyv+nEX779re/b0tZyoxwAEk2CAyb8q4AwsGCcfNkHoHEnxE2AvfezvPmlClvd5j8Wc1LToxh3ntr2nofp9s0LhiDOQKQAAIcAB3/ryOcABOyFPHZf45rwQqd9Pj5489zEDgsHve7YcJU/s/O8O3t29//cx19eAKSIJas3dXUTW3k2oSuHqaqmj0CiP3XGYZQbbWcTtm5bG80EKSmkiUxhUdvrfaRj3XrvfuQcoiIBCD5tWLA/+szXEgcKOXUTHttE5f+DQB631GP99//13Et+FtzSitI/h+3NHLeHt/+/l/lYsYQA3Eya702s9baTN8JHdx6600DgFj885YGsN7PnA/Km1O+feGBh/dOQswI5r0NNm2bXGQfIpKc3QGRx3WMAf93t3RIdmQpyULP/dgPvwCAnhm07u/fdzd3Nw/JS3fglIVpUMfNA4CIJZVl3e6PfEkcAJAAkcMgtJ3nOWJxvYa1NoPBIhl3cVf1MLPeUl9v3QI5y0V8JSJkHL9zrcc5Vtxa85yzZnN3RyTjX835NoA2AIBwU1ObCV0lVXcgGl2NY38ZAMgPAGgfezXgshGMigq/ff3y9e2+ZAoFrSz850f1dPuq/ds9R/v4gwoxIzOj9m5du9V2tnq22gYhiDhab6MWGtxm9ddx1m4WgKMjlsnb/p7QZIK/NmgRLNZp/JJmQVmKATMxeR90E2L+vCH6WdUpFSoc/WRo9v39eTYHpvDeDox0KshCeXMAkO8A0D9Gd2CVweJDfns8Ho/bmhl08LyeHzVk+0r25Z7i/EAtKYmklHjQiWuby2t95hkUFwymM8Xy134OaO7iLaC3PaFWuRStuXgW4kSC4b0eL0Bkmc8uQPTuMQStZKo2hBG9B8mSMoWe0Q/98f46uiFjWD8pPJmBrGUbl9Z3ANAR4byZMCcWlttt227bknn0wF3reUbaePV1SdE++r6UUnJZihzHeZzHeTRT1W7dLtArbGpTBhVXe4wIO8Bs4yfytqPWnUc5HGgBHCjZRovpeLEkziIJbSZjDojMKqzz+3X3CEocwtGjn9w/3l9nM2AK6wTWMxKlPEVj8h0A7GNvCly2GKKEnJdB280A2s5WWzUPl9vqwcxQ+4uXdd2WVS3vr9f+eu2vNpIhN7gKVR9EgxhQTG/x2s/a1ANHGz9ltApW93ceHG8kR06BnEgQrNfjWRbhXNaFWq3VvTdDJBEz01GINCckwoQEAaoB0ffX6+wOjCMPSinnIqnkjNeCfT+bBeeVylKWpZRPXqaHnq/jte8sIxdBU9PTVNfb/XZTB3s9Pz6eHx8fbUAM4RD+i8yGAaBqXbX3eB1n6zY35YywVWahSfNllGQjwuMMv5wL5e1+5+OF3qOfhiRdkrn2ehz1OGFKHeQqIPt5jI4lhoExcdooy3rbbnhdWt5aNZAMsm7rtq3repUiPbTuH+8f78u2bXJbN96Pvbf9daxvR+0O5PvHz5/vP3/+7Bd7b9IjPUa3G1FVi/bWYp9ccrjk3+EWARGcppKDUzYfEQ7rdX/h4pzXx5f0Ad7I+2nEkpK5aavnfrwOKmURSks5o+t51rP11rUbMIQ7AkCmDLLev3zhK8JhZgqcJd9u9/vtfrvN2xoM9Hz9/P79x8N4TdvXr/Ljh77Oj+8/16OqAQnsz/cf379//96nGhLdpgSdmYiZWLVrb6XBXyI8YOapUVLOg05X0jLeqesMP6UH5+3xLZPrQdFPY0lZ1Ux7PfbX88UbSFAqm/fo5/P1bOP+Rp681Mh5A1kev/3tc8Hjw4zL4/H29ni83Sed0Fr0+nr/84/fG68h29f/SKR7tI8//o+tdg+SjPvHz+9//v7HH52FhIRZraupWgyhLSftvefcyhXhAETilFJSHzo2mg4fvWzdgiSRzC29aFDeHt9KtDNh6Gkiqaua91aP/fnxkUCKoyxbP0PP5/uPOlI95AhT62p5U5D18e0fCS5yKUzUMGL0AtpVoA7uY7tQMGIhGv4DMTiHJ9WuFkjsExy1CKQQJJxoxWVlMDJuR5K0rKPuHcBePWkWBnge+1JKYv94jhs9PkkdpSzLsqycBMO0xnjg6JJ0RgQQcUoZIBAwEDgAEJCSIIRpO+1CPODydQDGsHa8thFf1+Pn82iGUhJjuNbDugPn9X6WrQiFVmpOebk7dhwI8ugUBAQiIxEhDyQ0JwSUvLR6iuSUE4Wae8Ssbt1Ne92Fwfrpv//581UVRJjHIzwE+R2w5EzeoJ/dUZZgWUfj+9SgvAXn6pO+jyge7pG2jLr/JGMAkDsARLeR7/mg6S9l/BPm58fz6A5c8qTZXPyXnrcio3ftlFZH0UvbQVNIdK0f0yTLIUleu/Y+WvQ8FgyAV22iXF+E1uvLf35/36uBMA0CFIx/NgiIibwb9e6UgDKXJRNYP3pwBsprv24GYAAAQNkS9IMn4nEDgKgdQsO6hrYzp5xGEzi87fvRDCWnmahZN+C89pClJHKt3gPTgmmZhZyZ8EDXcIJSIePZSJSyhbnZeKHDzTwCAK9aTOkksHa+tnh+PF9VQS6/DqBUVgsUc4hwC3ALTJydkiQKa65BmdOmvZ2tVoLZJ0aSkqAfdj4/QTwndMew7jrMCngqDUNrq82ASxIE03pac5S8OlHKQtFRzSlRWrvO1BIv8RxdagGZ8hLCGC/z5PPZhAoQCSPclbCC9XMvBY7jOKoByzBOiCDJKyDPi127IxARwdC/gYUCMCMA6L4fQmF9fvHMjKj7OSUANwBwDOsY1rVPCeo0+gjXwanFxBj+uaUdExAThgYFUIaA0FrPygQw3A+WhVS7dVWd/EmZvBTCfpznie6fWxoHCUuxWq+SUoLeelMF4VmbAkkJ4rzUo4J7P3ri0SMdrWpTYGFJLKyvkhisE0+iLZtZd/VBPbwBgIf1cUYnTfrivcMgggNzYgrTKjooOVJiKq1hag9IjyMJoXtetm3bto3aEFC38XcI81T6yvl8PdFazKLz89JCsEkyw3Gi4BNtAUyBkpe+C1mPdtQlM8pSCk7OqxfmvJSl2E8hsF6RUymlLIXO0/Q8z+rXpWWDCG+9mo56GhGBgGYzjpkTI1hvbO7AKMVH1W3mhTCVkrO+RoNR07o97vf7nWqt9Wyn4NA6Uyq5lJxL3ougtyPUPCLwivBkK0LAgOoYSYRGszdIULKZZfCK3o8jKCgt2wattQhrSpnzdttuNtYrJKmsy7YuRFb78f58GgDICgBWExO4tlMn7Wzm8phSThlFBvnVGhoisARib62HajMafNBVE2O49V6W7fZ4e7zxUY8zncQ42ZCyzN7Lk8HqTmHmI7EeD6nbpZ8wSinnPBIXHD47TEMsrS2R93pIdpSy3UMwFKzXHJTX++PN0a3VI7Hksm63bUM7QY+P798VAOSfAOA/39/fn6/jbH1cJo44xGupLHkppSxJcsopCc09D127qtpEVHojs6BULHgrCb2fzIN9VQZsbg7GnCyAfoGCQEPzdj1kaKqjd8+plGUpy9tWBKwecJFpzqZBqaywjGdRonULJIlLwBjNMZVbj2VZ13UpOSYYJgAA8jsA+MfHx/PjtZ9Ve1dVn6xPkbys67qsyzarGUaLkSlfO3q8J40xDEhKoCxFyPsJZKZOAuxT7KSSBv1jaKbCfbyUGLOZgKAIQ7lMktdtXbfbNuAng9nePpoFSdloyUyhjaB3dWCBsUkrR3eSxYLKkNcm/58Lfr5er9frOOcZNg8k4iRp2bbttm3b7eKwTORR+9RrBQyDC8awICkoJYmg9zCKCEAmULMIM6WUzXyYxyCED1U6XQSFAAgal1cgpbJs99t9WRdBa3ufRndxNg1MRTknQbcGYGoBJCQEbv1E6I5pAUpTXZ78Knbxc8H7vh/7fpx9CPB85s05l/V2e9zu9zvMGtcgTGtrzUdBBRhu1okADSiRqCMSetc6kB4mah0MrDcqXT1gZOODd0BDR4dXKoLhboQjwrf72yMnEbDqfEmIWhs7KTELhoZBDFOauaUxsDum4LQOgZOI/c8IX+rZPuRSHgHIIikv6+3xeHs83nw2ewLcejvP8/Le4IhhBEMQKDzEeuHdnUQk0ZCOYlivWLpaIDLPbD+QJOdcEk7tqoGbESIAS1m2x5cvRARg0T9rHO0aJIUUASnUcKCnE+1UDCNzEsqr8cAZCf9lwf8EgKiT6D7QXfOhD0q5LNvt/vbly5cv2kZvwAbkeexXu3Jk/QhxKRqx967ee6dcUFByQVMM6xXWQbH+hXMDSV6WdUFTV3W1MDUmAKCU19v97dt0WohLtI9uFpRIxkM9Dt9sShC4hikhUoJpKTcIOP8e4ZjE0W4w7+BP8ch6u799/fr1Wz9rrRWjo1tv5/4cLTLACDcECJcklEQSnyeY9/PEFTkhlzV6GwtufVCxP7c0cCrrtm1ofaRlrqpEGMiSl+3x5av13rr1rhgXOyEARSBUVcO6QWJBYBZEcDCkkVAmSZ/60P+5YJ3qnFnYBY4Ip7Kst8fbl2+//Vb342AcNlW9nvsLc0oJkGEs2C2ISUouiUPR+/lC4OEKYjOriforwvgrwtvjjr333pS79T6ItzRki9/qeVi3erYppZtSOSJsNSy0NfSEjCQpYNyivBCmsiwlhtjD/vWW9vFA+Lg1CJAQkNZ1KTnJ8Dka3Eof7+0g36jyRGoSEQE4gDIPuxN31dbOHVlSKn14JfTWatQ68aZAkrxsepsfaMwI4E6XhG+cqHWjsE7h2q8tzcIJkAUHSccMRYauJ+bNRyQplWVZ3czV3Dwva21NTQFAvgBATD14HxogZklZsiT0Xo+SEyGcg4t07PtR1ZGTjDSx5PlghVtHN22yP1+v/TgrDpqE2+vj4/U6jhr1PI5jfy2nYlofFuPJ29YM4T6i/vm5jB5lmkBexCYHIBqXahBJXjAlyZKSwEhNkEtZShJGCkAgp1gHULYYAMhXAIhWW2tMSHm61I4iHqzXcxeG8Hocxzkw9yHQlzyA+HzhQ64Qpk14fw17KZRB3rH9Y3wDUc/z2NdlaQqyKvJjWcu6lHUxN2O6FuwXLMCSdDjNMTlMZhMisQcgJWRRs2kRxwIIgQDIOQ+xmDMiejCuFiR5Ga2WrwDg9ayVCYHXmVm5m1uo93owDVngNNDttasDp1RyKUtZ8kSjY9yQTHjsx7GfZx1vj6nur9dr388a53kcx2tZzCCtkJaeprucuSr9RY76lwiPBI8oJnMlxnoBGWmaZ/DoV14c32HPJYQUiBThuAZKKtvDri3tZ0qMGC7r7X67325bb7XX7i3aSQjWWzvPWs/zbKZqjvyJquXBCUGPMAJEhPM4zqOeFRBnb3rf9/04TqjneexlKaAoi6x3vyrlbn1498VlnDrBepl7mvByAgMinjXWKGDxsin4bEBOtx5ECIqA4EDOy1brrwjvIoTgntb729vb29v92PcDoblVAvDe6jTJqoMzB5RSLqUsy5pNlTQc7JLe1bOeZzsrYLiptnaO01ChnsdSypIlIMkQEk70qwv/6xmGQYhMfWgPiXxCqkA8YANiJMZPuu/n/X3ZI04sLDAEJffWe/9csCVmCDct2/3L12/fvr59fLyzG7rWAXQcbTq0jP4+MuYhy1yzNuojJ5zAX539RAB31V5rHcZJFc7zPEopqfCErK/6F/rgRw426LjsR4TTjDDNNL4Di89McIgsJ0wKMQPLdEmRPg0bOE/e2+eWFhzmL2W9v/32t7/97euPQtAreccwbWfOl9rchIRZSMaWXtfcEWMYkE3q0aioW4Mw662m3AeBukGtR8k5pyhJlmUt+UK9vH3+5sOq1+FfbmniC9fseMEkNN2k8CrGJI2EA2fCAUDTJC5mBgKfESYAV+2trPcv3/7+j//4rbDrntB6uDYRkT4husgpJQROucwtTQBhQ64x+pdDyNU7uKpIktSngxrUsxw550QMaXk8buvlBWfpM8LxLxFO/S+GoG6qHWUgYcijyVlwdmVdpr8FjvR4kNrosodEmkKtBQBMdVwT5bZkBh83MXEuceFvs5rD2Z0AYBl+LqQzFXFgIM6L6ei1dLi8f3Q+8/h2f9yWzCOtHJXoqMvoSmzMgTgVWwSs7u/fl2N/7a/XUT97HzMTgrAW2s99RjjCeX4IACCmJB8CAJlJSEh+LdgsAhA539YhbhqNo7ROfw8WhIgwZ0cIN4gYviKpQRuEI3XgwTXR6WQCNC+QqWUzuG3btpU0o9UbXTpHQhyUEHUgyQ5ZUOvrZ6F6HsdxnGefQrXZtgEIjU5MPP8IcNzb9Ok6QzzvFAeZbrz4ueBZACe5LYlCTz67AUpe6HIdhPBwMx7YbbjNBQuO9fbpB0xEbjEa49fDOGkADsvkdn0u2EerhYgmUqQBKAkoJbDzldkHql5r7wMAIxp3UoRqDCLP3LnXIxE0+8XpolzEtPwvCABSAMDDh7ySx4Ib1W5BnJdpncg87bg5EDwi8NeCW+tduyowZUlJ0tVyiU9vFxveEJDHKbtoOjxRSuT5Z7SZA3EmFgGtT4quvWvrvQ/P6OkLBQDDV8PUFSe/9zrLRnkZHjxDEduaLwMo758RdggiFslUSqLQ02sfjXihmcaM9UofX6Qh0LAauCKsnRKnUpZSpvBmon0eF/UZZgnNV4QvnS8BIYK79e4OJMiJBa1S9OPCMd3C4tcZRoihd+11lveJpu6107qt67qp1/M86nmePv5E018LRmCWlBcQEQpFO0eE4wLGaaxXedhdRgBJEpYk1KYoUYDzst3WbSTXQ281cD6AmefCAGrmghGGcfzY0mHaGwYQMgQl0NP7+TGf5ukyCTDhIYCwNlJ7GHzuwlPQXel2v9/u6nDu+2vfj91ut9tta+p0belAZEm51EBEAjNqbXS7fTDFGMNMh4frfDYoiUhKiXpvXbsqIaf19njchw0Mol/K24n/DYa4m9nYkAjBTDxIafPRocsVgMBqP0cPEhGBAgPjl+FngLbj+XruT5gmUvkYX8DBj7ezqgO/Xs+P5/P5tMf9cTb1oCvCMWhiSzP3SSJWAxTgwOFPQK6mXZjIIczNHGX4HtFVIgtwWm6Pr2+T+kDetXXtrTOOaxPmYz4jDOApeGANiOGuvY8kQxjdNTzc+WJbA8El/xs2UFqP18/3j3eYfjfl9Xodr9dr5y9n646U94/39/f393fdj9rUYDDicUR4dmwvNr0FIANyXD0XJJZk2ZDMwPwvtiSXy80QpJayXMIbJ2GxnvulwAkENwjrPP597obohgg66a7OU/YIOrzZ+sUpjWktjIMvExGDZ9gbDI6tUDuP1/P5fDKMAiI9n8+Pj+fzqQOtqo3hEmq1s7VaWw1AIEBBMyNDA4ch7oSmjlIo9W6jJ7hs27oupRQkJCbmJU9exkyPB5MXAhO4WYSDXb4JsiyOQlKQQr0TvvazaSCnWSoMZ17Hiwc0vl0MADBHZEoenFdHLlvMITOSy8xvKLQdieAYuo3Ft21dllIKAYA8Zz181vOsn/iejp7HZdAaAyvkvFhT7dpU87aty7KUPBwGhHManmhs87bCwb5iMTdT66bTWrAVDUyBKUeEa0S8XscwVeKpd5zb94KYBgIWER4WhMIWwHlFLrcz5iZg7V17yxmGk0Roq2cPTGts67qWUvKvCJ/7eR7ncVIpZSFKmXV8qw7D+94mMQE/zXjzuo4IEw2LecmMbv3EYWCjiuM2SQnUe23n8D1prdU1SEpQyoN0arrvZ9MAToN5jBhIjuSj4QQIcN357siUxHz4/vTeYBbDkEvvufTuFNYIrFnX7pQQt3UZhv2fEZ7WyQdvt+CEsvSZNc+SVnU6ZTAOAnNveTCNSkZiVhEd9JqGUXtttfZK233bMGFytHac+3F++nuRFAuS3DystdaO19kGIY2IxzeLSH4xhmBUSoNriSJZzYFRsnvY1WWLIYLK2Qi0hdXjyrtkXddlKfkvC94HXXLPPSgtlBZCCHAjBLfea28JCFMumWaUWirLsiylJGJmVVEaniNgRz3O4zxPfuuGaaXcwfr5+njul/O0pbIqUCquoW34jY8zPAS1l/vyX1Y8Ms/eNTinruYwSLaIcTFYJ7jelMGadeLBNyDK2/qXM/wBAP56vp6v1/NVnPLqmMroUJvh8ASqdaFEsmyrDIVYq0OIWkpiZlE1RWYMDaPX8dr3175LNZDFMDFYO1/v789JCehR1mZBKXcMq+frddY6jOsRkQARRyOH/AowwHBHaC0k567mMVNb8em10HIZ/0WI4VSUUxaUlMs2Vpw/Ly27LvCV0np3lGWQTJgw3Hqrx4kpKC23e6pnrTVXkZLLkksRZjE1NQDEMO/wfH28Pj5eH0lRlrtjYrB+PN+/v09lQ8ft3jRI8jndTE/t3QIpfaIUiOSIFNchHtXGWSum0towhi2lLEvxetZaz0rDoz7nmLCCLasvidKyreuyLOWvCx4zv94/brI+mlMqEOHGw7hLWz13WZxSub2Vs5xnTiI83ZzF2EzMbDjIedjr/efHz/f394R5uVenRODtfL3/+HEZX/DjqBaUilBYO18fdV5HeJWygIgUEfHXLd1qPU7MdfghUF632227+X7sxy4UuYwN5GHWem/tZshl0BPWpfzLlt7P1i2QKbSd+8citdZWz1prUwPiVNKQACsN845rRBJO5cGnpAQHC4kQEa3X4/W+lh8/P15H7TpNXWcx4LMKJxJxmWwaNzf1qa9IMExkEQkd0c1GE3EomMO0MYH7Pm7c/WxOeUNZz/Og0FDV3luVlIbHwqDWzlu6dkfOXgT68ZFg4jltiMw4o0ze3d57b90Cpgg7HIeFiU4BM1EMBLd34ujHRyH/+PnjeTS72nyTs6N69VzKmq52d2+tuWkLGciDzP4fKjOE99Q+IUnvaL2+sk6v8dO6YaZyr6+XMHgHt95EEGE0BvsA8UamVdVRMhQOPZ+ox+DsD0UQCkopRci1UjNVM8cpZZp++6pdYc7bgMEE7W0smL3tz4/X3gz48hLGUWz2bqOdZno1No9jj+79jEJcSinl6gv3gZrVX7O3vHurO7MOBcJ5EiLmgtjeM6E1grEHIoJYcun6a8GgaoGcOXH0A7U+p7jaZ30OkEaEg90t/IowxHC80t4V5qAWTCmXtmgPjn5QtL0ex3k0A7rWO4GxPiOs7hNGxyeFUvQzuAwO0mwYQQMw7fUayoOE7j0QAPQ8z/M8a72m5PTE4e1gcNNG4B4sKZfW9fMdDvAIFA7m6GD1lUbJHoHMjHOuALmG4SA6AF1GCjE5Hx0EcLSVc86lq7qEntH3j6a1t+7IbMP6GmFqiTWAky4QsyRCCq3kekIJKdvjyxtMLkQN763mOaWLaAJHbjbumnq2+8aUtvumDNqOzBA2IGSXlMva1eAzwtODmxCjayWanBpAzgmZ0zA3DtU2qIn0PyPcAZAYiSWlXLqqdY7ubWexYYr5fxlhwNm1FAqtO0U/YQPO29u3366RdadrqzX/NcKT59dmYtAdF8y3r18MtZ2vxOCmEKbdUi5LbcNLYpgeXLxFmA1VY0ImZEoOglKWxcNCwyKGcSv/2xnurSPynNyTci+qxhG9xbCIQSCYk8v+xxkOEpyeyax1T+T9hB5cbm+//SOmfeWhvZ5Hlr+e4fM8j+O4Ur9ORTHfvv7dvZ2vJRGEoXvvrLmstXW1z1saSwZCzsVb67XV4WkhLFJQHDivN9WmoU1dkojQv93SXXsjEg8gSSmrqrnjnLminMZ/Lq3dv93SyInn8Ao59/dEoSdocNnevv0jpn1o6u3YS/ocrEXg/Xy9ns/XOZ3NbLkr5tvXf0Q/nu9rZvAINCTSZT2G7zxc7sMJSFJZVwMNPff9SJMH6pItkFNB0KFBDASkSzQ7eUaD+ePXuBNiSVktols7j7OmkkuOLETMLhEhMu/cmFSuVIbRjdy3YWkKQCR52W4x547CMTgJfGUAFNbP/fn+fmrrXXu3Y3LX4batS07CMdtzlxdYFZh+WrQsy7osy9pzEgQ3HcP43E1bTacQujlyYYshsvM+OLSCCMQsEQQM3sJqqrU1C5JkqnP04YB4Cbik4m73r/dFoB9j0JtHJLUIBEagvNy+7NUfhfT4mQedd1TDI98Ruczf5yeYk6qpb5msPn8s8ON5KqT1gXPMQ357WyXqc9hLjQVPinNPMp+v2ZHV3tsp455BgTRauKERIkkiEBmROAFQEFhYIxrgN87u/WWaYtQFWAgIcb3fFgk94HPBw5FCKDit97eq9raQ7T+gXQKqixQwkbS/LBlkZm5bRjuf3wm+f5yKsj4GLpAkrcsq3p7jDP8nAFApw069MmFYrxlHXMC0CQtBDPUYyaeBiGTLwy2ZhsuRQZhFxDRmJnFV5Wn1NFz8hcfkrLKUwtBPuxacp7JHgPN6rxq6LqQH9NcvidzVQJ3rLaWU0kopk5LjcUto9SnxGeGLeyjCLNHs/FwwjlF7pZwIob2e55zt4dqlMUFEzkm4pIJnqx5aq5gDIA/YDQiH2bup+nDtQIkufE1XGJJE4bwsy7IIMzP0qBccV8YYtGxAebn1oC5MureXLEtZlmXBGWEeK/7rlr7q/9gy6snR4Pk8FdLyNr6UsnCYh9cBN84FT0rOAWG9nUeazsGA2hgR3FZgkLxsxOQ9tO4pAJGHwRGQU1Az67W1Tpe5SroczSPcAICA83a73240Rov2uBas4znrGTitGiTN3a27+yiIkNFjZDVTIfNrzVM6A1Ay2Rn9BbWeirLSRc/moTWv56dz6eeE6d1N67nnNAQpENqJANyGFWfZHozRyMeCicQjgAIjGNDA+nnuNeWUAUVQ5DKIGBJMAinb25e3L3H1kf91wVkVOK9OaTnPgSvW+9vZHDnz3NIs/x7iXw1gQqt9JzA1g0R527bbut02ej4/7GyTET8WnErKKeen9nYca0kWaADuoUODplwMJG8Ptp4wtO4JiTmZw2R3QjTwdr6ee1kXYCGhXwN6x3o5OG+Pr7/9Zvt+eOvHeZ1hI0m5NLWgFJjKdnx8ND0+3j++nM1Rcoe5pccZzn9Z8Zwhx+jmOjI6RKSU8X6/3W/3+x2/k57RXoMR/58AMJj+KaXc27k/l5LIICzCnQDcNfW0DsJ96jVRaD06sYzXZH78BOvn6/25eVDKKCFzdPAYDgnEwGV9fPv7f7R3jgZ6vD4hdU6p1EUNOHNaVF/QSPcff/x5dEcuq/IkMkn693dpikxkWDidrcKYd5PT2+PtcX97vAHb+eH1+b1frRZgxuiubT/OrgEkAOCjzp6DllP6tTvdVFt0tanCGXj51eYKuP4US8pFfVByAPAyYg6Y1FV1G4Kw8c/FkG3NnZvSNDS8Zi5P3GPqzXU2yCl4khRcIYadOYPkZV0ejzE/MuYEsdbhmpmmfbADXn/++eN5arBHcFA4SRpE+sdWEoU2rbW1rmp8OYXMruB4JHMuPc+pyciSSgDL/Dswr5lD6/5hpwYXwzzBD52Dlq8cW7U7pfXhmL5+/fb1y31bU2ICt165nsdylILT6vOC7OkzHDi8Nrfb7bYtWSguS5MImJM8JptWdX//eH8ePYgjiCiIOJdlW5fttg1JI04nrGEVMgfrjvUgc0ql6JLTIFqM4bgpB45OvQz9wku8a3ABWS7nwpKzDKKi+4CXndLqlNe3L29f3h7bwmn65vN5nsdSCh7nxfuY6rTPBQOnZb09HuuylkHq+fSP+VxwHSNajn3f97MHRwQROZGksm637baui1BYhdpa+4sXDHx+rgjbFWEgGazZSQRAXjODVSZwA0ZeTOeIpDzKApxVY2vdKQel9X5/PO5v99uKabjMNar1PI+S8ayTjTxAtMH2CHdjJMnb/fFlySWPCP/yrZkz0/o5kPhzGNwBO7ETkbPkZb3d749cciLQ5rW23rsOL5iYPa5ABBg638WvQeBIgiRqFpNQwyVzaAVnRBREhNmJ4JSTMF+Sy95qH4JG1e12u91u2xKJCV17xfM8j5ITnmfr6j5vjOsK+RXht69Zkgwfq8/4fkb4fL2/v7+/D7jUxhl2Ihpzvu9vbyIiFNZsbOkZYf9rhJFEUlFPOQ1rY0aSOSFzcDmSEFi1npIkTpJozHLkMe6EaJLtWqs6BI0Y67qs27ouPsxPe8N6HrnkjPUybb+GzQ8Mxm2c4fvjqxARz5mN/76lz9fP739+/7PP/iS782j+Syrr7fH4QogEYd7HIK2h9ZlrmWeYmCWbRrpM8YmGJmmsd9znoNaoLItwXpZ8HmcSJhzjP+Y1rL1VFxrK0DlGrvQ0VDoVzlLKkRL0Oh2u/+XS8iEsWNbb21carwN8Co7+ZUu///nPf/5TU0opSyb2ICL8jPDX8AgLjTpNVi+3H/i8pQFZkprBEJIiIE2MIybN5VJu+woCXLb7cozzDnRFeFxarQalMcVw+mymMTjaeoN6lpxSAu3abSQ+v84whBsjD0dknHyGK8J/3dL19f799//1v9uyLqsv6XLhnmN0tvvDVDXUPl+ly91o7um4nIncYZToCEDTD21g9QHWtbt17ZANOG+PbcD7bpguS6cxireDU1q227ZNwii7EIG7KQ4/30ymNnY0fXJIhrL2mtX2CFU18CFx/NzSY1b85CXHxQHUKd4eio4yHrPBX7s+F38QB+0IrXcL5BQ0SSoA7sYkPPlL4B4OQtm3bR2zDq3Xc38+n7QaIDuwpDIo4Jmsenula8H9z2eFfPutp/vj/rjfH6uZqasrEpG3TnicCmkFSfeC/fgpAbOteE6rmtFbEgCIqSxgH11D5fnqAFivR5bJI5ln4lpvzGl4g6nrOlwX5l8O88ABBvqkEQQREiXCbXj9QWirx+v5/kEWxNlhGIsziyF6rUhw+Vra92eLfPsGaVzbt2U2yIfnUnf33hUTyJJuGXX/GQ1mu6u+v87+bwvmT1byiJtdEXZtZyJwHkea5nLxk2Z6acYi3C2QB2d1MDwQFZHQbQzDgZSEJElatzHz1q3XY3++/+RASd2RxEdh3C8vpcn15Hi9GqQ75rSt67Zua7m8TbW1rq3XDgEgsoBsBfse9YkjvYr+/jr+R4RHjH1MjzT9vJKsVyZwlUGrxb+EOIYL7+gNXPfw8MSe8/NmpXoR1LCsBDzG34wRsGNLv/9IyHlRH6kKp6z1jFaP4zwuciv03iFjvucyaEopPDwsvO2g3o79ZBEe/vuCureXzDoZ9Pk8uwMl/pctLcw8pika4ZVGhfVK6NpTSTkH8i+/9Jkl6Cd+ebUN3c0xYoyqh4BwdXU1R+eEkrfbkkseEu1ej9fHx8/EednUgQFYxM1SNDuf7893mngxISKmTJBzLoNlPi/fI5Ss7e+vZVkkLcvCEa4tPObtQnbsRzPk7P8e4U/ADS3mFezawLWdy1IsgOOv6x3DDy6iDzIPApPpKH1suktNpFWNKS/AZXsUyZL+GuGcylbVgZDCU4Rze1l9/vjzz2vBaU5tzDklyUmSTNp7SK/obf/4eQ8BWe93brUOi7iLNtnGMNSx4HFL86WHGq/EJUGEsA5uLaW1ayCl+NzRkxSh7INf4yySAimJAoTHoBDHmHdq3bSbjHlM2yOzEDOhzTP8o5TtbOrIcLWudvL6/P7P/0Vz6ky5P6Dk2+OeedJrppld0PEia/v7d5AV0vr4Rq9ouj+fHyTMLCRjRiPwfJD+ektfCOOspgLCwrURczNHFotPdRBeEb6E+5KBBFkyRvgluhlU8W5dezdZmiGX7ZGRYGRuvR6v5/vPst6Ppg6ESAiECIWsPr//8/+Ncxr6+hvmSLdvv+Wryzz4qhjwkdDb/vFdFsW0Pr4RtFfff/75J8toIdEQQczptAoAF7t+ZArMQ0VoHja4SwBg07CfR3E4bMPnfy9mNMx+4sysyXHMz522ymqzsnSfv+oF+Sr3dh7767kNMjp/smsreQQA+dXQKBn/Qs7DgNBPdfeUYdBwvjX1kdIGX9aGAQDyAoB+nE0tkFMe9pFputk44FSHlTHZBnAMzG0drnQaCTE4nEUwrIeZWSAjMSspIIa7jx1hvR6vjyx5TDlHH82lVBJaff1M2EcemdI5lP7rnYfdc16/fLkVjn70a8E+eefP9zPS7Rus3759e2yZwoHHlYAzejjJhCPTegKA7rV1D2QYA06XXOvZTgQNHCb7KQuFNXR47vtRa+uQdA6aIyAIDCIksLAeEYHMIp0JB296gtPW6/EqQmU6mXi3AJZcGK2+BPQoZdhKnN0AU1lvqSx5yUtZb/ctk7eDrxM1m052vKqnW+S3t8fb25rZASiVrTtcHAtQ1XCblIcnAOhxDnotlXXbtnVb9uN1YBiOcWcpZxECg+jxeh3nWXvHaRYQSDiovcMWFj5pknZpY51tRFjbuWchKClJkiygU1gLaPUFWj/GUJDV2hXhvK5lXdZ1KUvJHH3nIUVC6L8+Ibd0123bti1TBHBa1JBnvkBO4RbW++eCbT+beiBjGbOTtudTKLQhEnHKJS+IGGod7dj342ytkw43LEAcc59gqpNjuLoLxRTXOQ9Wz7ijEoGXnHNJluja0oFWsR/Pcr/d7rdmYc2CJK/3clu3bb2tCycW8ubXnRl1vD7NETHlG9Iw+iZ3pFQcZXQTIMDBDFz70B6+AMD24b7IXLb729vbl1sRDKuMI9EryzKZSa7HcdRaWye98m1iSiQcXcNNuyUBQpYUOBesV6i1VSFwW0opvbhT9wCSXAytKjPL4+2tNguMbkCpbG293W737X5bBnzW+xSbodfzOOtxVixLyaWUMk3yw4FSQU7l8qRW1wFo+meEPy+tst7evn77+kgU2nYZ0GMuy2rWzbpqP8+znrV1njXiuD05i2NYeB+KJ+RU4MKY+ILjrY/m1bIuq1mE6HSMa2Ea4QBfjrNZoGCzQCmrbo/7/XF/u5eu3bTrVavgJMTuR3qDJLfH435RvRw4gaSl20i3vevM2/8S4bPpmOIzZid9odB2ZkagafbQmoEN68zaam1dBiw9jIVTKsnCOlivFVACORWAGKMH+EKsrROE1nOrm5oD5OvSimuix15HjSlNgaSoD7Xr21s+6nF69NMnmdr24bbyWiDd0+3b3367TrQDJcpDx2BqZtQq/WuE/aiDiTci/Ns/voLV41n+EmGMHtrO/Wyt9VZbF9Xr0mLJuWSdBroncf63BV9lmDZwa+dR25ggGBczzdXqse/nWbs5clpys6BUHB9fvnz5+uXLl/x8YnRvxxzrgvb8+Ph4fjyft3z3tH37z/88zuM40NyQURAQ5uhRhSr06wwjwHQLVPPMoeczUfv+rM7Lo/dlWQS9H12BM0jaru9xud1v9/vttqWchBB82ohTWrKAVTToTbsDJwcUKctmxESC3tXmKORL04WU8rLeav329b5ljg5DCp5Dhpm95uM8e3AJQwgMBMs5p8HMuPLii2RMnyAIi5haCiCWUtbRsgUAF0nZLSBxtJ2jP5/P03h5o544CVp18+Aii83htF3Lto5pPDQs5cyApADLwiygoCeYudkgkOTSe7MB18bnerdLyUVpWXtrvT/uj9vCYKAeQBLAoSdae4mqqlNhn4Wo9jomjKf0Cx8apGoeqhRCZBc3z8gpl227+7VglGTZHYDI2x79WGqtxgsVRUBAs4bT9viCTSwtS1nLsuSR/YZ6UAJKRSEAVGGmm8hCkq/jNAunMapwWS/FHpf555d1WRcBdXVAEkABrV73xGMIkZTZcITel95azmVOlSEcaijtPRgREIc1hEcoSS7r7b7/WjBLsggA5OjQj5TCI2hdwqybmarJpPWwT8BGZvMuOUTAeJ+EkofZsO7QmDI+5qn7ttZqa65qY4DQsl6uvclhQKljBDwo2mQ0Qag1IOApoEyTpOzUay6ttTYoDZ9YTW86h6LTHAgMJnlZb+dxBAAIAQDKdNR09NYREMcQY0lez1rdWi0oUpZlSbOOiqsUYTNXD7cYWgWAepyq9Tx9TL7mNDnfYMcuFD3UPK4Ix1DgAl2OWkgIoOARQEhhPkYx0LKti3BZYWpkuS+9tVYv2/6Lw69dETkCiWmYuaPnUrfaao0rwsFpNNh17jvfto3yst18f5I1q6+Q4Lzdb8uAbeboKiLCrh3V3GDGE1+hYOfracuyICXOsw4newqF0ucZLus1zs1Jhkb0KihnzxWwmrZWW8NHD1q43GHyranVXGoubY7R+dRcd6VZXDGOCfKwDMJBg1+XVkQAEtcardZ69i9fsPDy9hV+glew+qKBVXy54VWzwkgBolYER1ckkpRSZtQTtT5/WneghJQuMZQlDK0cpteW/iy7r8FdQy1satOiAq3Zeez7Di24OJUbDHq2YVt6K63mPNRd+GvBzBYAxHxVSwMPnAOXCYaZIxAxQ/e+v17PQ7E8aH37G6K1hFb3PLiAvz2uGTgwJ3gZE3jHMAKSUpaS9BTQ+vqphpQcePITUjZyrTuB2nVpzfIZoUwA5/V6vdCqKrMgCXMLrc+P93dwLvfgcgcbXkDYai6llfY5whsuDbLIVT6LJJZE00DfP4H4UbIDgvIlTR+1yhdo514yE4xO2bJu0zqC43OKjU4qB5FISiVd8w+HoTvLJz9Bx4Nt1ySaXz3X+dOX4tobI4QjGgACaK/n8Xq+w3Z2A04FO46/fnETchKZhgmTffDZQZ0fJvjVyJ7qUoQwCAcujmm5tX/8/W9fH1sRyGW712bx9liLgLWThY2F/aK5qjqgJAfMicC1WlMfzjdjBmDJAtasMev3nx+vvXbt5/6xZolzGPwcJw3odSn7fpzdgR3cOyLg8+N1VPXRpkKkUXO12s5jirpzGl2YoaYvq1qZc/hc3fqcyTRB9V8K8eFMEVwwLfemv32bC07LemvqeHtshdGqiIqJs8T1IA8/CCAQIXAFHbOEl5tPNUlGcO+B0Yfao1mv+0dmtH3MCzzrXHBZ6llrNyAeEKDbfuxH07iMDBDDtbfzOM/z7BYkOU/9yTSI6OZp/LnwwI4ASDLlE3+VxHtHCAchCfOIL29fPiPczJGX25YZrLGImbv8ct1UBxJACSICV8OmgZyXm9+2bVmWki84r76/fzyPs2s7X0nQ23OOJ5kLLkvRrl0NeNCptfez1rNp4LVeskElO47WmgFKHsNGws2BJBWLMUqIaECvDk455ZwJ8FMw7W7KhGHBc7TF43Z/3LZFIi1dR0VTioA1GK4GHm6u4430IEaeuL2BNwuUtBrctnVdSs7q1mtr7Xw+n6+9Nuvnzmj9/KhTm8eD3FqWUXQju3o/z3aevWtTG7R0IprKseN47apmQQKj4RquPmfe+gRgJ3fdFddigRz0a8GmnRHCYbp9p21Zt2UtyfMylCfDvcSaWxr33fxpZhCAMmZeDeb0iLAFzDNcArydx3Hs+77vR+3aTwbv9bV8KrfKsiylLGWCMsRg/dz3fR9K6hidZqSJm5z7cx9Vh3BJM8IBKBmAfMoZYPBjtEPfAjnFX7a0zpGvxGXwE3POJZUsntWBJS8jDTLXT2rHSI7N5rQTcu1qrm2e4UC6JH8K1o7n87kf53meZ7N+omvdl9wvmHYpY77EGPVOHIPV9/ERs5k1dWE4B/jurxfiMIi4zrA6ECckiUvO59prra2CBXKyv27plisjhiOX2/3+uN9kTpuwtCBJWrbZmTQwd4cAcHM1NXOhIUm3Fg7eW+vqJIFC27YtSym5ovXj+f7j1WprtXeNiF5TTknnlyaDDLaUwfkUDrB+7h8/fuLw8aVPqcPw1TxeT2YWIpZZnY4zjCzZYxBF3bWdx3keEMipWPxasJVTiCCMuWxvX75+e8NAjJlDSl5a76316lbHzMEA8ImSOggQZ8kKpmBaq0UgIwdPSXVm8Ha+fn7/GFdSN3cd5+yC8ueCy7Jsi6MAEXivr48ffw52dLraREPfW8/9+Uw5IZLknAZ73xwICDzCpouBaavH63gFUird5xl+AYAdA0La8ghJkhicBp9i766qHsgRkYSZICbXE8kJrPZGbK21qo6MjkaGTjgxnedrP8fUd0CGQaJHhLAIZE4AIDnlMjD4nEtZVt1q6xoO06k53GyMAb1mMFy8nOv/TMU8+PRcC3cAYslukRisHYkYAORPAIjjHBO08jZa825jlaPQM1WLGBbPyCNzjQACYgf3mJNHzd0cGM0NHRDQFV3bmT7en3vtNqpyDvdZGADRVUHOSTBpnR9XR05luXmAB0TM+evHq47xD/hJy5qfuOrGXzNagSQ7isSWydsOneAaXaK9O2WUfFtLFgrvtY0DdzXEhn/JpAEwQcB4x8F7095ab0PqS5zUlQwgwNW1MfFrZEuBhMQzBQyAAJQ06aNMJEQs6+zvhwNLXm+POcm7a2+1nse+t7N1c8AL9La5Fz8/dlkBeZAEcC6REnkDPfBaMESEU5JS5oQN1zZcT/ocW+VJEginxEOtDgGXUD+anvt+7DC6UpLIcE7XnDn68dqPEeHxxzAtNx3TspRlKYvAUIbTFeFhTnt/vIaO0j4jvLcx/QQvb6/PAMenFc7ndidBSm6BSN70RPhc8DVKOi+lJBlY6uv12l/1OhxlCUpSylQlgl9Ks2jQj4+P9w9c12UFkUJjfDKF6/hl6nmetTtcthw0b3yEVMbs2xn3wCvCSFLW4zzP5/P5Qd7ctNfzOPeXfkYYEMDJ7drH4T4nKcDEFQgpxdgGrm72uWDMuaCknEtO+Yrw6+Pj46PGZHBsQQk4r3lYcg14lpkldtDz+ePP73R/3DEF5092wLjyrA/pmE1DMSYe2BYqpGW7Px6Px/D+cIsrwiRlba219vN7Am/0K8L6bxHGv0Z4PBwwmI5jkiciRqtV24V4fAcAvK0olNetDAOlMa7s4+fPn+do+2I4pgU4b0VV1WIWnJIkJPR8fv/9/+BvBmkJLpMcQOg6nRLVTAfuOnqvpr0rQgwF6dcvX+QarnZFmMtkFm+JrO/k1xle7N/OMPw6w1PJ7ABj4imOUlbiFd3b/nr5dUuzgyyU1nuZJE3Xdu4fP7//eV7OgZQWQ87r2npDcIwAYskpB0M/nz9+/9/FQFYFzgEe5mzo2o7jOI6L+TcVC5K0ExG4Ryrb/cu3337jaZSoV4STzzs3g9U9f0Z4eXn711v6c7DOiHDXMbQNAJFJUk4pR/TD2/7+067OA6diQWm55QHcuPd27q+P95/HBbWMwVA0iS9uhrPnEIzez/3jZ1q22ixIzMXZaI782Y/XLMonBJRS1jYgqEh5WW/3ty/DC4u5L5O2IhPkjfp6DjWiaW/tPEr0SYS/fCsHJ+zTCsndpy8GyPgNiwsNMwmF6afF1wBLmm9G1Np0eNClYS55K+T1RV769CXL7gjMDiRluT2+vOTtvhahsItKPaT3xQ2mPR/NUSJ5KLgAnIfLQ5bee1cPdK0QvRW+CKG///7jefYgGeWPtujDy9wYhylrWcpItoiGntciIBw8WlivIsnf359HszFyQDIAcJ5vIU5PTDtbH98jy5gxvRb29vImQ2Th2iOIRJxI8np7fD3k7b5mwZh98oGSpWx+mSYRpzmPsw97LTDBsHbuaVxjHuAKrvUUuoaK//jjx/PowYkJwq23GJaDajgmOUkZqlNEJAYkuajc7mEjHfXX63k0B8ZfEU6jQ0Merm5qtTa1AR8PXWIS9ubtRfNxtwWROZkjcV62x1np7bYWofjsGw+kahBeR77y1wUjRqhgaDvlcnwPdHXtyARXwvP+8/0zwuCm3adLpjMgyyCyZBFCQEZiHwNSzWE+QxBo53nUZnPBeW7p2aCJcfDrrwiXZVu3FQCi1nFWIALCiTipOaPk5Xa2Tvf7WgTjczLuAA7HeomI+HPBbbJSiDGsnTRIzIAAoQAQOKYAqQ6r5qMHwxVhH9OH1KegYim55MSDIkgRHsYdYoA+I2my3ps2G7ylseBLHDQho97O9hnhvN62232OSOo20mAEZ06tmwdJXremDtu2ZsG45AFD5uKX5IT/suBE45tDodB+hAkR8hgzPpKSGUbt9ay19iD8dYZHlh9X96LklJJc3oAAYQ3cxtiYUfsMeMxxnOG5pWVG2N36dM5U/8W1fjvO3duxH/2iuYNILqoeJHm5qQOUpRQhv85wABJHjEGjg/ZxXVp1uOs4MLq18J6mSwm5zqW2PnysBurujAOG7BxjhtOopCSXtSSRK8KICGgYpqMhPPzKFIfHiMBft/QgiF8T5WeEATnldXu8feGo3l7v7+fgrongEAs6kJRVHQkkpXRt6esMz+7h8MX4XDDDmEsXjGFhnVPKkhMSuI6RE5Nn0OpoBgMTDzvyfs1Xh0kAWoswCzEijkkTaGHKGG7tHCCwCkuazsx/ubREmBzCVVs9au2qHjHO8P3tKzSO+vr5x54kS5IslMvS1TxQ8mIgyYnHkC0b2W0ABQCy+Wx5yF8XHO5mjhgKiJiyFkAOjN7O49iP86xDYHrp7K5naZhHqml4IEsu2zX5ec6GIjbXPvHr4/V6vV6aS1kKyjjDGwDwupSBdsLl8DoY3qPdONyFbJhjm5h06TKc2EmsVQ1KAWMS0XQsDWTiGIVpkBALsTAJgXe32jVQiuOUuwQMoJyHWlaSKsGYqyhAwgNoFgLXiFEeIE4DpsMvz18arTOy8zhb9xgHVHtTHg44K8GcqMXrwOpjSjgvEHjYbLd6HuU4x0BHjDlQhZgIwuxm3dRRiqmadTVDYiIUGvbYEcHTfAADvAWMIXFOCfNVyMlobCfJjsiSy/h1TElSHs/i+HV0DAIYbtFaGb3nQZklnrLeAVbqmP8gzMQuuSzrdts+F0zLMhjqg0Y3vDORLt+ueuZ8nK2rj5/nSB3H5GvVc5D9hXuo1Vprm3dx4kmTj0shA11toEUQAJhzXIUd5zKaRA5EkspShDC0N+KUl3XZlnxJqwMQgACIQytGPz+bk7NT79PVFT/t1plTWbbb/cYAIA8AwFwm2Akwt9YvqpG2Vo901DlmNcBGm2u8i/24bKxJwdrx2o91XUFIlgTz81kGg3s76nnCOJVCV53DeUpngThl7bowhPYmJFKWddu2PA1MdFimDHMN9H5crgiSPp1hp3VQ+qS9j7FC98fnZEtMYzBCTNkCycREx2tWE/NxjVmd80HCwd20tSPnkjNLxorW9o+P192AC6alXByyaS+Lps3a/nq+eCnLQnlJE6JwSllySklQRoJVBhorPAaY3G+5tdY8tCHzGMJGYd5PIsk5p5xTHtBqa5THpFtJU5P/Oej914LneLC5py+u+JCxaa9CdJytmwWiu5u5mw0/xHPbVgNBKbCD1eP5412Bi1Na1skduhRx2E70dnz8+Jnud0qYt2WOUzca3A4RjkHsWWLoh2Zwbo98juFCHQEJiYXD1SMceHrx5mGNd1babtsGMsVYxBwy5yj9WjBenkAjD/wF84ebNmHEzwiDz3G7w9TkGLNcUZZgsLZ//PwOUnpgWrZrABNc6tLJ1f9jcUwrpfU2WVCGgzCS5Lp4ivV67vkzwo9M4EphjRAZkBOrD/iYBy5WyrGf+3HsB7+9deCC6XOEwLWl5TrDl15yKiLHGR6oUZj2igCf890jbJj9m/Z2HstuIMVAigtYO54//uCydae0bnTNoBqW2kGTq//PDdNmmLeHzuGX014qTVgUaen13Jc0I7zdHhm8NwxtTBxInNhDa61n42VdlnXt/fV6vV7P10tOHb/SpKwRzC09IjzH8cZF6o8p4B+padh4xP38RFbCrdVWu/ZWz1JWkmVzlGIM3s/X+4/1XtVRyjqvMw6boLGA1ePjxx+1bM0prfcxXrozCHMaeMz4tuXYX2spadIObvd8zaJ0mXTHFtbOfT9paa2bqn48nx8fHx/P5JTW5iSfFULKZdm2210AQAwAoH92GEai2j+ez9d+1qpTepC1a3Ch3M6TKcYgt2mmNVnTJLks2/1RH7cpw4qh7AudKrP9jx/PszmlfLkZGFhvZ22QU/IEDDbNm88fe4d8+xr3x9t9LVe1ERGuHQHcpRtlTKtSLjmXnGjCNlv+7W/f3u7bkm21AE7Zv325byXRkAA4APhMX2sfPAa15+u578dZ1a1rSylFQDCl0AFq6gUKm9q0cWBJed1ub+1zwY4OEAPVb6321+8/Po7uONH3JNLDez2OA5dcAsjBfBAp6vPVIG0u2+1+W4vgZ8YXSgCuJuGYUwTgOAxCUoBS2e7p27evb/et5MGCyMXf3h5rGa5VI8LejmPf92PvMTf3ax/DtNVUOovM9J/Zx7C9dgl+/lLtc1qW7X70+21dEhPGtMP3egxv49ePn8+jO32+nGPE7/F6oa6BJAHT9623Vjvkm2xlXdd1JEUzcbMertrls4rBKS1KQLKs2tOXL1++3NclhyNJWVa/3/4twtbO5/Pj+fFs0+Mv9mM/juOsqjNXybkQl5xhtO94vlk+ke+xpfO63U593LaShQDCg8IJ2/naX6/X/nw+n0dzHEmGJPkEhNEDiFOADfrGoeEBSTYfyjuhS94YoeHWO8uCKZelrDiDFELJ3N3y43F/3LclA7CkZa2xLtv6uWADAGvH6+fPnz9/DiiCAI/zOM/jrDbdzmndOHHetmHHlHh6HvrU8gAgSVqW7d78cVtH+yJo0BmHZPbj43Ucx9mDBnaZkowxssfrAwFIkjlaP1778/UyEZHEaRQTwuNwjCahGxKRgFDe7vcHXm0/QQQAhLzdttttWzJRSqX3HuNLw/jc0tbP1/v373/+2XhWHeewuZjThgDRKC+UtwfPya9Dw+KGF0I5InxrCo/buoxN6AAIiO18vf/8+ePns7fePyOcZM40Pl7vhCS52Oj7f7y/v/u6bpLWbZ3NA7LpCRvuA6CV5JhuX75+w1prrQTXrBYqy7os67IkZi9maiGcWEatcG3p4/n+/fd//t5kGtuMkUmt2vRQBchbcNnexPp57onJEcIdbfrTADKnZdm6weO2lsyEU6+G0Y7Xx4/vf37/GEIR+nVpIYwtPbI/i+HL/P37d/gSac23L29wofgxQ2yzcSSrUd7e/vYPPPb9YHCVoelKZdoE50jgHuEwx2YH/HVLv3//4//4Xy1JFsmS2nSntTmPI2TrwXl75F6PvSS+Gh0wtzQiSsqrqtNtm1v6gsjr8fr4+efvv38wIhHhle+n6aL/+qCUy9otwPrxev/+x+8YvEG6ff2bT0b69SrF1RKVN8O8ffnbf+LzI3FYp1SWMXRujvYQALpsHuDTg0M6AOjoNT8/WkmWMgAOtUqAXdr9iww1RQrb3SbQMJxBtFWcMhEb0j/tU+wHMDplg4/NJETl8s8fk3aOQ0YtBgiurZ77C+99TN21huHq2sdYW8ngboaOzGlUuWC9JRk2GswppUQYBm4XS4JwQN42uJYNAHrrfQ5sZ065lMutfuriA+LxtrId7yAfrw7p5uwTNUtrptDz9Y77PtiA4dYZIRLQ3Euc13t3GAeSEIfnr8Dr9+8/P15HU5z3wNTpfg7WiTDrfVg/W1BaFPpFC/nyuC1Z8NPt2a0PnLlddEsekgghm897AICcAKCtT7emoeBcyqUcvZyhYFkXthMbn2eHdOPF59nmNXH0+hI865B7gVvHcEsTeEHgvHYDvgGMASSFoZ9PsPzn9x8fr7MqDSEkTn/4+Eyr3FXbGMenwWkJ1lnmy5fHtmSmXz1x0wYQrnyxP8ZRLpkHa9D/NcJqHoHXGMzr6A4EZDpo29lfaG6QeZlvXwSOCAtiG+NeMdz68MYUEmYglLI6SjkvEUuW0AO08vuPEWH+v4twa+dxns2A0kp5DIuJ4C+P25oFf9E8rCOAa6PL/WFwztSSXdSAXxFuauY+sPOcl/W6nMeYDiRyd7PuBkwkqfAnfRiXzKEnOqqamgVCaIRpE0likkCQ84qS12pzirEw6Gn1hc+Pj4/X2VRsPm10FS0jwDHm6p3nfroGJRIb2EMEf7lPt6xrMxiCu3bBi+623m7bTX10u/49wqpus4Wby7Jezy9dA6bqeR79PE9fliK5rCVMTU01UqLo6H0YwXhgOIQREWeTHIAEkkHy2nqf5SBgdKuIvr9e++uoeh2eudwx8Wys2Hqr9dgrQBClCyYLoOsMXxE2hDBj5pgz6dr9/nh0u0iR14LnGf6cwTaFaDPBGsApi3x8YLPj490eD1rT7e0Rc/adEVOot+MyA0TwMEAAVssRyIycuZiZXnnyZGmansd5HrX17OGXywzitaP9stc6j73xwCUYLj7DY57hX/QdN1IivMYC1MeX1g1Iwv/PImzmPtKHXJb1CqwM8U5KBfvTzvc/utEC6fb1N5hEWHUYYMvsIRGOLRcu5g7IYsg88NNaz3a2M9U6hkdckz1+RfhXh+jXils9j6PmTJxTyogBGAC8bbclj+F1k4o2Lmew/TiP4ziOL3VMfoO/XsFSAaDPKXtB1wghmUYlw3wppUz9xXa+/95oeUDevv4Dro5It26qXecor4l6hRk7ACJPcJiZfRRNcoLVfr5er3MAPH2+f7+Ah/i8pW2QO/YWJJSWZaXR9kdaBrb8K8JT5h/99dpf+/7aT7VALgv9y4ITAOCyDXUpv40R6fdhnDgqMIKw7pjK7e3U9tvbrTBo8zG/52we7jh91YVnj1lVO2k7l3Is5RP5OOtRj3qe57EfZxtUREQWW0ticL1IZyxwOWw4kKS8qOTEYFqDAAIhkPo1qGBIY4dXmbu7qgNJWiAzWN3f6Qy/LBWvZtraLYCk8ON2f9zv99snhxWndkBB1nt3bl+/3hcBPWxOlm+ESMhyNTto2Be31rCVVHIulyk0R71OV621Ng1AppAI30oidL1IZ5xg+k5pjOGRkVgYrGmf8Udc1lpqa832o3YHYoyIQeQJlBwkWyJvO3nBQMCJ20kGAF81gFJZeVgZ3bfZK5ijNRRQQZa7U9b77b5I9KOf+3kcx9E/RXpDS4KAYdrqWT2fw1Vlij8ZahtDTMcIE/XBzUOAdUkMofWoY7al4F95/dkCEyIMAGg0voJqXUtrtfl5NjUgMR8WGkP6jVJKRmu7tzzG0E2RR4HZPUll/f+0dyVJruQ2FCDBSZL/7/uf0L1zdymTI0AvAKoqfATb3NRGESWITJKJNz29SkYfxcI4HBxrqAWUxcUHp5yyh1mH5WfMlJNDitlb+1nRilYr2z7gT7vkeP4NbUYvOVFJWFLwnxkGdAE0Pph1SW90kc9Wa7kAWHrPo48BY0zzLVIATjOZSDaRkyHDsBHnv0ktIqqVbj6XXHIpWclZYOCnto6zi4/J5H3we+523/d9X/d8bBeA8oPO5Qi3zNHrtbwC/+67YAunOeAFHlREn+HZveI5ntAczJego40+jDmH8BjzcGaxF93m9cKzHQlqHi4jIFmoPMtAwECq2PoUrNrwMYY3YnoC3fyR3RAQHlOtzuSoI+es13Vf13XN7UNGig8SJQCJzfA1j27ZWq8eDzH2W02l5x6lrDOMlhIbbEkzb3CEnmOsyJNHa4dTjqWXPsYYXgREwNFyILLmYDsUvIn2eKsxRwwfQHx7Smvyml63mRjPfWaBqCMEOnLonYNh7ItLM5qu4UIWpPwkvTbiQuA1W7072DhL6mNFowoG753p1kJMwYGssftQ2oEuaREGcOi37Al7AM96LRFmYYFezBTzcydUKlzfoDaKYXSR0XrnlNQ++0NqAYqbRU2AI4UYyE59GJsnyBqdIsWQQoJa77bHbPd1vb+ur2vGMgUpPmjxXDyVgzN6vZpec9XYw3vn0QBA9oEChY2AjhQXouBRFizbpQlJSWPrRD1OmR54tPdizaL4bAjhoJfOnuGNhBRyyvceMu7rWioDY022VK7vBnTgvOZbeKch9GJGsXPOgYIuxJxxLw+y+n3XWlvr3TKDYwxu7i24raeHB3djPuewmrsw80Z0Tt9GQ0wl5+Ccc8B7qTWnJ/yYrcPHoE3WHK1NfVFnOBTL6DUXcZ/OC1JIOZdScKv72ThomQMA+hMAwCjG2xutKCzhJYv1QB1jynJuC88B1/u6rut91742UmQpKYVziMw5+lhA6QWUuj6wa5KRaFEsIVDpD96hCzGV56Oo2STI1hs9Osu+DcEs9mX8/fWura+PZ9WHRctbZDnv3BiMlCCgGqWX4r2jEGLuiqeSwx8F6/BH535QrqO2YzdReI2+1cT2utXwK8Iu2ZhwmvLVxwLK4PNznE7ggeg/+Dc4s4b0FHJ5vJ7+pP8s2Z88BwohRN4811qzv6/rbmN+F2w9cWFRq0HkxUgY2Gc9a7InCiGm3NUAT9u49E8AOK1BoRRTiilGW8iLwTpTa+81yZPc91Wv+77HWht9dKipGWazOUfvCwh8Xjz0KtaMNEzBmQJDjHSutjSP5z9ebs01N8s0woqRWygGWLJ67+aLbLFcIiJwxA5s33AjIFIApBPTEUIIMeXS8bu18D3DsrdsssCHdBJf5GzxIILOOcf1vut933VtESRHviib1aySR29mw4bzvu47Bo/HuN/vI3bQOwSjmRv/htH7ZuB1XlK996pYEuTZ6n1rs+/nDGu+PAscTYf17EMwbW7WenPp59/+KNhUMOezqffeRut9R9V3o8DCjYBc71rrXavCqgRBUc0fBVt+Bq33+52Cx63m57r3b9h7sxJiLT/s+foF1YFMkKmvNM4mOIQwQWa7v97XMLTv55LWbctkvZLVmCGdWJIcQ0yplNrWj5CzU7AutmCPe2qt1V5bg5JLKYweTOC2mr5tNrQ4h5jzt6J1jd5bIZ9yzpn/zmrOfiTidOQ7KhLavD2FVB7/+LUdyHSfGf4EWITYVaL21zXHGnOs/5hhFoFlZERBQsrPZ9GiU44x5VwfJ1ET1EDsT+0c6405lPJ4lFLKmUd8PV9TgKKwyBKW2RR1aj7o+Z5LTuEIPNccvYVE+fl6vSRHDbtUQCxFo/Vs7K16BFYleXm+fgnwJLdthtWhlQKFGJRv//7rSztK/LNgm2G7r04XEob8/P2w22JKKbfcWmut1uZApvwoWIdlQTzKfdXrvurlf/2aAj4JG1Y+emutt9qDOEIf06OkFMl7Yx6P3gtQev3x+48dg4fNq0cLRgmKCm3sl3dbpttnSbPMQR5kaavHwWeGPfDs99e/vnT1/njdt0OJ99JQrRHyEym9fj+zeVL33HrpvdcrkQeZukvPnwXjnHOGtcyncAw/zRhN+yif31cMScBD2fv0ZcR2HQr75BUYW9r7o1NYypk5Zufew7EYM/tm/B5Gy1z75Czv03bWP6cPb00TlQWcQUzMbKkJ+h0d/I+N/xf83z7+DRGKQbk6wMnLAAAAAElFTkSuQmCC " > </div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Even if we try a smarter and attack <strong>both</strong> models at the same time, we can't succeed at a consistent rate. Be warned, it will succeed sometimes, just not consistently.</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">loss</span><span class="p">(</span><span class="n">outputs</span><span class="p">):</span> <span class="n">entropy1</span> <span class="o">=</span> <span class="n">Categorical</span><span class="p">(</span><span class="n">logits</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="o">.</span><span class="n">entropy</span><span class="p">()</span> <span class="n">entropy2</span> <span class="o">=</span> <span class="n">Categorical</span><span class="p">(</span><span class="n">logits</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span><span class="o">.</span><span class="n">entropy</span><span class="p">()</span> <span class="n">kl_loss1</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;batchmean&#39;</span><span class="p">)</span> <span class="n">kl_loss2</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;batchmean&#39;</span><span class="p">)</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">mse_loss</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="n">loss</span> <span class="o">=</span> <span class="n">entropy1</span> <span class="o">+</span> <span class="n">entropy2</span> <span class="o">+</span> <span class="n">kl_loss1</span> <span class="o">+</span> <span class="n">kl_loss2</span> <span class="o">+</span> <span class="n">distance</span> <span class="k">return</span> <span class="n">loss</span> <span class="n">\_</span> <span class="o">=</span> <span class="n">attack</span><span class="p">(</span><span class="n">loss</span><span class="p">)</span> </pre></div> </div> </div> </div> <div class="output_wrapper"> <div class="output"> <div class="output_area"> <div class="output_subarea output_stream output_stdout output_text"> <pre>MNIST Model says : This is a 1 with probability 11.50% MNIST Model 2 says : This is a 7 with probability 11.48% KL-divergence is 0.474844753742218 0.47643253207206726 ATTACK FAILED </pre> </div> </div> <div class="output_area"> <div class="output_png output_subarea "> <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAPAAAADwCAAAAAAbPrZOAABnvElEQVR4nOz925IkOZIlCPINgIiomUdkddcu0f7zfufU9nRlRriZqggAvuwDQ9QjquYLdladKIkyMsPcWBlg8OWcw/j/BgCUIlJEBD3cw8NxfSAiAiLCPdzdHWuptZZav//Iz/M4jv04jr0yMxMzEyIhEnpfn7Y+/s//zD8f//4///1//vv//PffiQiJCPvX9/fX99f38/fffv/9t99//03/4z/+4z/+4//zH//x+O3Hj99+/Pjt0Wqtrdb2z//4P/7jP/7j//iPfz2O/FOZ8m8mIiYisuvq13VdXd3MzA2PYz/2Yz8IAAj+b/b5/xv8/+sfCQDIS+ruf73DkHcYIjCvcbi7g5EpIaKqmXvE+jn4lw8ABER4BAAiEa1/dv9XJkIEiHCMgFj/OhExEyKE23/54fkrupspq3kE5E/Nf7p+UwSA9U/uvwkAAB1RmBAiHABAFAAwwtiYGCM8IsIB8O92uJmpqYVkeCuvr6+v5zXMAwCJiVlW0MIIh4CI6UFckUspzBhugVzq/pjH3oTCRmdmYgBCkrp5kDwa+Xih6h9fz2sY5FcAETZDZ2eWP38+u4G0o1VGt4HGRMxM74/PqY5csayQi6VKIR+Bb4PdM8JhREQEOESkc3nFQFfVOeeM/O/M1/P1fJ1DIwKRiEVY8n9xX+fFPZCRTYSFIMwDuWz7tH1vQmCziwQBEiCX6gFcWiWfp/fxx9fzmh5ESIgB7moZ/L//fHVD2bQWxtARSsTERIyUQT/MLZCB4T6nSEDgY/7yMBoiESIGREAEeHiER2ARkSKAbnP2MbvlO4I0ruu6rmHr+DALL4sNPMxMIwBQICAP8fJw2y3aXguFje4eghSALA2QS2PiGNZp/PH1uoYBEeVtcEs3xOvr1Q3KFsyMoaDM6znK34wQAyCACYiZiJnAXd3c7G0wIAJiHuMACABbLqJaaw0gNp29934p5P8P55hjjqkOAEjEwrJsBge3qTNw/Q55w8LWkQ6st4c9AJA9kAoASRsR7hrh/evreU2D9/13U1czs+s8u4E0RAT0cKD7/U2fERIhExKRiKS7xriGWx/xNvht6P0xM1c347apA7K7zdHP65wRDhERtj4Q6WCW+1S7YZiO4SLIIsIZ8SIskMsWyLLlHaYARDIBZEAqZjrHHHOOeT1fr2tYvOObzZn/2xxzGMom4R5h4fj2bn49RCJFhKWUWkuttcLrhTF9vPxtcP5CHivSIaiaqZrxbg7EFq46rvN8dXdzd/MIgAgICgBCImERYRFmQ3DX0QPycpYwNw9zC+ISwCJVhMImAhKyexATi0f4Cy3G+TrPq1+3h/NIa+9Xv65u7h4g5Kb5S8J9d9dlRW4VGbi0rbW2tW2Ln+R6+njZ22B3t3BzQAQERJiqOlWV1QFZapjN0a/X8zJVU1ODO0qsoMVyG8wEYTq7Iwdy3Zqpoto60sSlkhAj2AhEIvEAJAQAwEDrPl9fP8/0ZNAdWXz28/V6vV6AjExMMDAsdHTIU0zvB6U4cEGu256fIzjmSfZXg9/nM99RxKFzzjmnBCCXZu46Rz/P50vnnDrnpJLxjFfQygvDwswIYTp6SAmS0o45EXwFLWBxQwTAMHckYfMIXC8KWn/GeP3852luZg5ESAgQjtrP5/f31zeVJo2kMIFh6LgcCRmRMiIDYgUugVzacRyPx+P4iJhnQR/nBAAxAAgPQAIkvB/egHBDRHA3naOXq/fexxjTPJAEiDifc3NTVZ2TAgAzR3Ez1RmqZuaxUgadMyIACQkwAH+FjID0EjPmKRFhdCRy4sKZidgYo/feOwUCkzsjSXEgCkBEAALMP8h0/9pF1cwMAIlLbRv/ClrIjABAK15jHaMUmQMLoWsnOF9nH1MdmAv8Jb6F6Zy9X6VYteIOAaqqqmahOuccwnOMMfoYCrHMy9wIWJjuzAIjHBFI2v5Qq2aq6qpFCMMmYZ9TzSLAXRkx3FCw5OsXECtfA0DkVii0g7mbmU7D13Cqxw/Xt8FMjMzE+YAgQB19XNIJCoNNAj/Pq4+pFisTYV/PhKnO0a8ibuYBAJBfq2qozjnkLwYjIBDgO4FjYSZEhMiMxxGptF0d2lSdU+esQuA6AfqYZh4QYTgDrACKIGYACvfw9V4CiRAo2HBTM53Kz+FcD6d3lEZCllKkEEIafPXSWYhCGFzB59WvPqZ6UKm11lJ1GeE6x+hFODJyI2R8NwtVnTKE5siPEiJGXhxCwpURrjic+SyXph7cxhxjDmEWArfpMcZUc8+rFiEuK8mFu+yFlQ8jIobaQFM1nVPlHE7Nqfp9h1GQy9Zq45VCQylFmImMCH26Uu/9GtMsqGT0G+f5IlAwnbOLMLlFACLR/aSFThFmxjnzyzEiIkAEWkGKhZkIIV0MDggk1YFK66P3Lh2JCMPAvI+ZdYO7hrsWLLJt29ZwqppO0/ctc/cwzwOtOmdRdapYj3fiQYFc277vcv9LaS/iBAB3hRizjzHVg+v28fHx8XF9fVNYD1MdIkJ455j0Fw9Pob8azMwMHAiIRHejIp+kACdwBC6OJO26+lWEswbycLB1hyHCItzIC0j7eDweNOacc+jMug/D55yqU/NpnWNWgKBa0rLbYCnb8fEQgMAACFm3i83N3dzynVIjKvvjx2+//XZWDh2cHmZGzKKNmd53mHUKMcFcZ9rEIxAJABCZSZaHEQIDw5ECSYCl6naeIkyYSb2B232kwTO6lQ1ke/z222/Ue36fkOcE9AIL7f2ay+CW72WRt8GZIBwfnyXDaEAaDIBT3eecOi1jL1LZHr/927/925PBxovDbA5iBMgiVYTfUdqUBP9isLvAekeIMjdb5V9+zeAADFyaea9F1oMEbuaqY2geaXAHBHBD2R6//Y9/4+vqV796D8jcb6KNsP56jnwmxr5tVOq2bQgA4gCAQFza/visGeIj8nsPAAwNG/1uiHlQacfHb//275vN69kYw00nE2VCzW+D1UIVGQjibXAAIjsg3kf6rg0gYj0PyAwQ0O5WwIRwcJ1zqpp7QERgthVR2uPHP/6nnOd1Xud1BQSEQ3QVCh3n9zRVnX1Mx4J1//ggAJADAHhvbX2lYeHu3qcDlQZZbUUEGkA4gOvo57MVPs8JZf9hW2uttta2UgX08l7yrx8zAB3MZtGZFUEgFSpta4QAaoCYAd+dINYfDAQMHOeVZxhISIqbjt4HBfidLZSjVQaf3aY6kJQsZ9S1v66hDiSMYZMggkikVrV4G7xtrQoTwV0ETTVgaZnJhrsHRhBChI3r1YRxvGbIbviotdZaW2UkUBsIo48++hgQYD51yOodTGQBkrYf4WZuZtDq1qqFU3h4eASs/HCeV+9D1YAQIAD0KkLgnrUCI5WjFQab3YY5IBccoTbHHP1cBhO4IpgacSlt/MVg2rdaChNCuNpUVTMHKiSMCL7K7zBEcJ39VZjCrwllp6qlllJKLeBuM3MbHTrnBA81niy6KhEsLUjq/vAxTMcYtm1T1QHQzS3M7S4B7LyuMaf5qm/RqhCAKQOzkDDXfSsErt3UAoiB3MBmv67erzEdqBCGgfMwkVLHnH8zuFURylsz5pwRAFQkBCDcTTXCnRAgbFxFMBTNQfb2iGzil2J9mI7eu5uZmSu6s5LkO6WmypsBl7Z/TDDQfp7jmGoWiGjqampGaSD61fNIM7EIF3YhDJ+zk4gUKdyOVvJIuweQkOgEn/18XdmXIAEIcwNSKW0bU7PjcRtchJkiTOcYY+TsgLBEuLmrupsRQoTOzug2CjJKzWYWC4sM8q7X8/kEBwePQGQkZEJzUzMzmZa5cs8w+n1NNQ8gwnUG9N2I62MZjFJradUZw3R0Jqm11FLasRVGn91WLQAdw+b1+r5MTR1JfPXmRm1/MfgBALi3WosQRrjp7NdgESYRtogw0zndlZeHGUJnb63VUlurzMRMzJd30PP7j58EiJkzUxYJ7mZubmU6kNT9wfME7c+fr2kegCKgI5MH4mzJxZh6B626bdsWGK6zC3MptbVW29bWkabVATgJbPbX9+URHkCYnQpzaW3r/a8exr21KquQnaNfvQAySyvu7mo6immm+a4Dw+Y4Hw+qsn88jixkmb77E/T6+uOfTEJMWX7l2+huHu4+Lai0/QNPBu2vry+LAKQiMPvsY4whxMwkBPN+zEnqdjz2fB9qYc4voG2lFAabYMIIxMyFwGY/v68s+QjDw+ZU5W3PyueXwVttd9CyOfp1OTFQaS3Mbc42hilnN9sm2OxFFFuU/cfvn0hACIT4EtDr+1//u0iRIlUwG0fhEWER4XEfaa8c1l9ffzoAEdcC4xrXuEaXbI4RqrlmC0Dqdnx8gOvsZxXmUuu2H9vGxAw+3QKIkUsRivQwrUPnFjZHH7wf5y+DKYuMcNMJOmbG6EwTpUadtbbZZkQgIPEUZpYseFhqbRsSECLS2FqrtRTJnMGAMmkNcHfInAHWjOEeUEDc/b3VInBiVmICs3y6ylQzX8kZM4uUtm37fmwAEGYQZTYLIA43szlGHwyISCLrZwMJgmm/ngIAMgAAw23OMoqOa2gglyLMzETB2f1ryLJtYwwVYmJi+vzcm6DPzkTBQCTt+Lw0sGKsooWFmYT5jkglU4FxDQsq2+OCj8/HvtXCkUmaGWKYA0HcBkM+w87Ps6sDl9Jq2/b9aJkwqEnb2thUbaUqq4sqpZBUnWqKPz4aWf9Cehvspiwi4nMOc+QiIszEaXCdm3GpqnOqr8IOj8fRBEMHCwMyorRH1yDZ1yOkRJJD3cztx+BCGKbjGgZctsfk43HsW0uDxcTuG3BnJmaBWSvz8+zTkUut27Zt+771YTZG77L1faqaX9ddQSKxlFJlNddx3xrbBfo2GEzTF2FmFsCwLKZgkVKaWsnfwJ2ydse2b+nhEgEEQNIODSrt0fvVr+7hWNqxH/s+zuu8rgtJGMJnv4YGle1hsu/73qpwMEtW8Ra2bHU1N7t7IZIepl8eBgMbr/PF+5jTPOJvHhYpZdX2gMJMfukFb4M1ZxWclQMySR5pRGYpNVtx+aHV2aylFkHXEYAUgSTtCCrb47fn9/NJMRxRtsfH5+fH9Xw9X0IBsjw8DbhsBrXtrbVaxFmMxczBw6bq1GWwek42rVx9qMPbw0ezDjbOr2/KujHi9nBWqVJqDpyY8p2YZ/wyOLu6iVVAQiQRESYiIJGiZg54t6wBAQOBmUnQ510BkTSnsj1+nH/+WSkmR1DZHj/+8ftvr6/vKgjmt8GmQWUD2kqrpdbCwCzmZu4Ypn2MuXJ6DQh3d6tjjunApaaH99YZbJzff9BUMweAt4cBM/IUkcJFBMboffQx/G3wGq4isbCAMJfsqBMwFzOPWAkB4915zH8rpiOyBSCWoLLNOcZeyedJjli2x2//+B//47tVwfCpwhg+xxUGVICLSikiIgx5h90V3ea4rmFmam4aq8tb1dUc6e3h+mKw8fr+E6e6AwK+7/A60rWV1kprNZ5PmN6/n/Y2eHU6g2stQHDbywTMUszhHgsL5QwvB4RuZkpsHgFEWDY3d63s8/qmCJLt8eMf//7/+FmFwnSMQtmiBwOuXDYnIWZiARZjcfcJYbNfr25mambq4W5uujk4BDDX1tq270f9ErBxfv0J6uGARP3tYSJmqXVr+75t+xZ/4jzt+v5DAUB6PodZnsnmQYJc7qAFLO4OEFkE1sLh4OHhOsac7tOZizkgieTUIijG+VXJAUt7/Pi3f/9/boJgc5yQz1IvCEgkuK4SIQV7hlTCNPj6m8GmOpCACJlqq23b9r02Buvn9x9gAUDMdPX+l6BVSm3H43E8HnvQfJH1r3/Od8cjgIAwQDKG8A1TCDCd/er9HKsI5NX3DldVB5IQQnDTcUOdIoDrdnz+Bh/HVihWM8x8DSTm6HzHCkgkAupq5I7sTZoHEiC7+/buTASGA6L0NsYYA+8nCFlK2/bjkXNNd6+COZoEREBCmEGyHZ86YWE84v4FpNVaiwjTGm+56ejXeZ6yoFyMOWLM/AlIUBghbJLlrB1Dgcr++NHhx9EEXa+rjzTZVeccLJJoDqIM/A53q36MMVUtABHZASLattUiTNlvCQButfU+BmRtgUSltm1/fPxABIzw0CIUiqFrBBw4DMv2YWhvDyMyExNLLaWUsppNGBE2Rz9fr+caeAtlInnHbCISxnCbyDekxYLL9rg0Po8mGPPqPe3IMYWwOAMgIJNBuLtF2tvHyKp/zRMREepqPuGNSeDWrtauBreHWUrbjuPzc0FpfDJhaOiINfOmYVg2S1sTxQP39G9NBO95ALjr7Nf5/M6eGxMTMnL2Y5mZSIQJ3HB9D4RgQGV7zIDPRxPyeV19ZAvOTHUysRdEZyIOj7CclK2XYww1u8eJTFhqKXU1MN3dnFtrvY+BY6plUlXbfjw+PjOfDR8Q6K4AHhEBEKUbyga8+dvDgiy1liLCQsJviyNMR79ez6976rz650ylSAGikt1jDc9LgYgaXLbpFMexFfR59ZFHetlLFEDEgMzu4KZTRx+9j97HNaZq3hXO1oKIiBABeI6KuG3XdrUGQ839ngvuj48fa65vq5du7qtvW9WxoGwat8EElC+6JB7mDa2CRDq8nl+IkCXSPemvrSEDSUVECHP9u4edBFomn9fdrkkPExEgcwASW3Z5Z++j995HH31OM4+Ms1nD0F2Yqk5V6Ve72hg03x4ubTseH58RHm5uOGeozjHzAcVQCCyykgfJBnj+a7vc7wQTAWYDZPTr9f0duFqoJcvdsjlQAZKS2S7EnaqBBVVAaSFFBH1GVt9p8STMgUYA0mobzt776FcfPaObRwBxqbXVsnAq2ZOfY05urW39ajimucft4ePjM4sOV7giJw+e6VEY8z2pf3sYWWrbDoE1PryfpTCd/Xw9vxZ8C/JSlVoMqTiQFI8E2sKdm2pQIWkaSEjoU/vbw5qTFeJyG5x2LA/3rnOqugcRS23b1gABMLJtonP2wW3rvY9BU9UD0lX78rCbuUboCB3nM9E3AN4allZbw7sBQL9K8gUFhAA0RAr3dSFu4JavGaGZ3+DAuMF7DpBtXWSSVWwo+B2mlSm7dJnsA2KiBWxBBQDvOEDEUmrbtg1hwZcZIVx1lN6vs9bi5zWmrn8J73qhtW2YDUZwnaPeMxAgKdu2EwDIlkErtIMOBoBAACi1lFILBrK07RjqYWlwrVnkbntjsHEaIAAQyRpMe9w/A3Jo6f46rz76mMrIHhF3apdgvYgFoihNNd9iQpRSSq2tYT6m4eDGlJ28qzDB9fP7+bqGms5+Pr//3OT5OocB1zbaHHOO2lpt27bte62FCML8NhgEXcH6eWNrsbXWHJASSjbV8uBGYG35p9TCYAMmLShc6ARTVVtNHIIwNTWz8zyva4w5mWx9IbEunGe9D4ws7mFXvy4hCEh7a0OAQAgwN52EEDa7MKLXr+/n2fs0Hf18frVK19W7AdecoNU2atta2/Z9qyWTBXwbDBRT8QbjAeK2TwskCSRpUz08LHEFLWdJVYgYLJRLKUJUxMHU5hgj+44MsOLqfJ3XNfqYquwevq7BDcd0j0DmvEjnWZggDERqehgDEADMdDBliGOC0PJ8PV/XmGazv55bEZxzqgJVG23UNkdtWVltOxdhBNe3wfdfbnADCx9qgCwGyKVuDnj/dthaa1ttDcEBTAObAyFJM8/OfxeREoIU7jbHmOM8r+sac6pwXlV4o5HzTAckeJDxWRgxbKaHa22rAAedo2SMmwPBbcp5nuc11HRc57MyhbuHB6PPOcaotda2tW3f941J6G8e9qk2dcwZa2aJZoFUihGwVAdiv/tDW2ttq1tLULNqOHABLs00+8JnLTUA08Nz9N5f15WIJ5V1pP+rwdlFk8LCBGFzxPsOrxxWRxchBNdJEaade7+unh4+qxCuzIclM/NRc6i5bfuehUpkm7YBQIS5Xtd1+WpqkDuy1GaIXAJJSiIO3XFrW9vq1qz3MOvDgBOVoZ3CtZ9PbRFEDhCJZrvO67x6H8OL2gpaeaTtfaRJWq2tFSZ0n6N7erg1XNMLvWphgvAY4DZ7SajMzDssBK6yMJY4Wq9t9PtI71sgBEA+y+lhHTH76/m0hT0kQJayTWNkAeJaff1+uG1t21rbBoeCjXNyaQ4kjRIg+3qaB7IEQJjNcV2vs1/XNcaMxLnDygHfHvZAzm5zQwTT3otLqaXW2tYzBXrWBYGI0MnCOY4yNZ09sVyttUDhSnO2X0d627fdMwZF3AZbJ9f+/PrTFpoekbhuQz2QCkuxLac07rjloGM7QTvYOHupakClkVD47Oe3A7GYr8qjn6+r57MUWereFruRLZORStsfx8cGYXNcV7F1ots91Zut5h12NyKie2JlTkMIQmc/DkcBqjxaG2O00Vptbdv2Td00ZwEAIgCAsq6MASESEG21MIRNzLwjCzagCGR07a7jen6/zmssqIzO/HakbKMVQtdBft3vrzkgi0MRIlwh0swMLZEjGnMwEZJf04Hr9rC9Mvi8QBYyIlvrtW0KCJi1MRNAQNnathWhlcXo9Dwxux/7nv0DMAigzKmEAACkts0swBYylT4/9ipgA9Kvbvl4ASBaWEfifr7O8xzqpjpGv5pNR2mHoRShUDB/ndfV59TMeIFgTd3dzFRMFRKnOsdKKUZ7voZT3cOlMOhlI/sOUN5lkcbiji0YfALlaq1FGMImU0wHLs3xOPIJpohgCMyBOAMASGnTHdDzcSA+9vyKb1yx5wQpk/2IiBjj6gvsMufoVw11lKbBhEgxbWjGqqkWSAwksFVhgkj3KiuoTp1zTAVwneOqfXTDcrAjYUwbeCPygViktu0Y980nKVJESlkTeRHCcB0YakDFkY9jb60wUbal8K8eLnWLQIq7zm+tVgYb/OYCMCESE2QusYBfcyYUb/SrggZKDaruHm6RcMXEKwIJcWBZ9zDtZQZdHsbIYWgxN6fKGSPV3bZE5AcgSaltn2xqpgFA0uqWkT1rSGEMUwhQBy4oZd/fHobVmloexmIeiCzB6yPMwmhzhlu4hUkIEpKA2ryufl0rajuq6ui1njSduFGZU+c0nXN1bVQDEJgBsIgIQtjt4pgLlh1uU0SKJMK7ko+EgcyhDsTFCZlL3aby5AnhEVTqsR/7IasOT9jwdINwJOGytQQnERJAgBi+PYxSHBClQDauJNHWYA7rAFkAERALoI3z+Xw+YzXvOT1chD1QqLifHSy0Xz0BolPv5JpI+C8eJo7bw2Grk1JKKaVI9RNs6HWeMwBZ7PawGvGAcAQgafvHx8eHgGeRh4hhnr1EZgKsW213BxAWlSs97OJAxLWiLGD7nWdkAm3hQMaILAE+Xl9fP/+ENXXGvMPCBQEFEYHRRsz+OhMArEpJpeJsEv3lDsd9h+0e8ez7RrXsu+fPeH7ZQuQzsUhV88SrIQRJ2z9+/PhN1q/qiz8HKCTEwlxrKa3wYqjdBtPqS7MUVVz3n3WOqcPmnQs5sgQQS6CN8/uPf/0Ty+pwJrqU2RftgcEGhfbnqWtwKoTERQqtv9XNTNmU4o7StiCe+BnUqO6fYZ1Br+dPR5a6aQBSDn0Aw5UQEhL44x//EF3j2Tu+UhTgUmspq7NMAACBaG8PZ02PLFiLlFIKd4qwsDljGUySiXCYjn6+nt/UzIEEkuDAhFoLMnKVzAFt5UGmRvl9Vr5Zb4uASLEQa1PDwwMcUMruyDUWfuELS21jmgeQlAgkxAhTJuLStuPjx+8y+5yDMCAwws2IBYilVknYiq+Zc0YtEQBwX5QhGlVKLZXXEEDzSEf4pNDZa42vU7F9OPF6/qpUAes+So4mijyf3ag9oM0xxhxhd/0bFuZhYXW1tPBOQRYOz0H79WJ09z/+/H5efb6HFWhABalstTAjAsysUmslQ3CbvQcEMjmQCNoMu1iIhDjnFrBoAmmwheu4esc6StWiMmafY840ODwcwUYXETgvxfqgbcWXKkgI6gOlcClSivTRnRqW/bouRrf1N+XdVVdXzSGuYELY3O3meOq4GMNm/Pnz63UONc94P9iDihT3IolV5q1trdZaUCn7IHccQAL0qQMWZ5xx0bH+8ixhgkZPHLWqVpM5015dDC4Pm0hECGaGlTe9u/Zs7q7ung1cEXH34FaP/iqMYXP1DgPCc9iteSVF8aYP2T1z0ZHzNv/+/n6efbqpqc4xGIEZEXCBjZy2rdXaSsGJ2U1mYRJmwdVqssXf4rthTr88TAslC3XOqaol3wtVu4FHlkci+YSVfiHwcIw5bIwB2cKXbHgU5lkYXQdlrQAQ4Kpjjjk028nG/jeDwWFSwuni+Xq+rq72nr/JYuwugKTifaQjq+R+FSDkUivonD6nDrp7gisJ5bfBRuA6r9czakvimS4egd24Ik+Ih60Wz9bynWGGFzlof73eRNvWNixta8oYOoTutl3AghV1Q2IpU/mXh/Oag2Lo7KX4dZ3nNaZbxrVRkKjUWmrJ1q7ifaQ9Z1v9Wjj3DbqrzatfNzQXb6IvvquldaS/YapOU6s32DNRfBH+JqR9fPBePz4+ZI1eAn2AXc8vI0yYwMcHFmqPD0fXcQndTbtss/arn5bz27y/vthcEBGgYbOzUCQEeXl4jFG4UGn7tpdkkE/YtqwZbPVkL5IgLtsRMYfP6/XMsIyAUvK2vQ0GCtdxnc9oquq6sJHm2ZqFiJhLooKMN2yPf/xDMJsRZuME7c8/ZyJhiQ3Lwe3jH+46rir0DlrgNme/rpezlFJ1pZjrSCcMIRMl9Glzqs77SI9ag8t2PB6SvcER6eFSJy0PFwuU2o6wjj6v589Y81LM9K3U95GG9PDr6VPT1iTChN21WGg/z/N8nczbJ9bH7/9eACEQQMdLQPvzz4E50GeU3al9/A6JJX97OJaHX6+QUusaoL6PNNwQfg8IX7/F4vmNXjfITKOEqY7efd3hEkx5h5s6ctl2GJIG+516LexMTh4wj/QtfGCmSswRgEwg8X6WsvE/Xd0DiEVul9h6OHqS+JDXRIBQatv2x3XxGsSGa7aiW2EEmwMTuV4j7/ACmkRg3PM/21uVVULnE3a/ZKvdCUuYwdcIAImDWURKqXa/7ovA7Ag3AiAJRyy+mICeeMhM9N3NfMo9uMz51/T8LmI8k5VojitImOocc3RylPYYjrhKsCAqpbZ9L9tWKeZpqoECXP2O0mtugr6yNN22fWsL7tSLED6/n8/X6zyv3m/iR9qNabo7oJTt0Fg0d3PI/ED4bXCKQDCLEWdyfkdc0NWNLYv3iZRA1MGrPhzP8+pDzT05hQDrEHYxlO0wFFhc2WCpdduvi4oUigkzIqiIB3g4eEQwJYA45qo8Squ1FQK31YF/fX8vg3NCZylaAPg+poRcN3Ok932Rkux1BADJcLIAmcir6cSUmRTkX0w4S5ZRmBCWOWiBIW8PL4MhPTzH6O5YNkNp2ftFCCmtjT56ICKGKt1JdQKDwmPNYiVG5qWDM7sBt9UIen09n8/zfOMo7pHer2uJKKUFSNFVvfituPHLw7iqVcjRcziQlNZagzHmmERQJH0MCZufg9ZPG6/zusZcHk5xgznnGAMcpYFsD888K7zUlEowMzdXi1IKlSIFfXVS16Suxj0/xaVnEKaDMExvD/fex5xvfRMETMKROyFXJyr1bss4MRJjVk23hxOMjIsf5ZBkxA17H0wIfhfKQJjvAqwf15+vqw9V84XmC1PVMXtHR0FppivVc5stO5yzjzF0DmvAKG3b8B5sbOsT/Tqvfl4XrGGamw4M13l+fz+fr+yWzSSr/dXDHh4kASLbPleCrJi92KyW/naHgSkZ6EFc2n48sCyyS+LwRCI7wnPA0DHnnD2P9Oo3AwLqOtIEWKQCgLolsU0X2ux6vUBjnhMkULbjwesNiv049uPYDzjP13me52mrTxw2IdxGOb+/n6/zPEf2B/NIv+8whEcgA3l1G2NNJxAD3z2tv91huBn3gFLa/vigLCnDytI0cFrEiMg7NvrzzF7sAhgA3XdYFvtjAcRtmi2JiCfH5JhX5+pU2vGD79f/8fh4fDw+HvB6PV+v5+uV1QaEG4bbED6/ns/X63WNvMPL3MWHz+cHhQMCMlvrYwxYDx7AgjwsNR/myKlTOBBJbftBq0ExRaSUIuJMSYzwMfros/fkkCVmZhG2Mz2ayFJLLXXOJK1kohruEvOi0H42DZR2fNxgaf9cH3g+v5/P1mofow8wCAM3JaJzhazZF13B3z7OM+2BhEBI2HvrpfRS7mQCAECeAOB5KFNTISFyOl7o86J+jX51dcwRUvHHXinmJaAOAlI3BB+DMWLhuOQfv//4fBx7q0JgEKaa4jiFdeFR+7BAqRtsrTCGa45BbvSKm4ImDMIX53mYAAKxsK9GGVcGG9fzp/38ej7Ps/d2Y0uzvQWhqgpcic3VDFz/YvDVEwO0DMbQDqH9hZo/PoCLA5Xm21bI5wUEAMwAFj6vi8GBpBSpUn//7cfHx7G1ikmkoTURRY8w13kTK+pGreakGiI87sASqR9guvBWOuecBinZU7y22VSVCoWN87vY1/f363VevW9pb9zjIY9858tUAA9LnNbfPBy3wT7B9SqFVmPMkQuytC2KFAq9TFb/Okz7WQgjSOrWWtt++/Hb5+PYtxoBFnCnQcTuls2YayYfibdamMJ1vRSBRBjuNkH/kkmqzuHInAY3VVM1Kgw2zsL6/fX9fC1NhvSwTrW5OhaM5DwiFPyXwXFeY1oA8t0lCQ2l7I7k84pUSJp55gxTe62NudWGc5xNGB1I2r4fx/H58ePz49hbWeM9I2ZmJAKbWaT0YYFcvWy1MILpSk2Wh99HOqHSqnNOpxJAJDWmTlUzKgQ2LgK936kxp7oHQNIJ5+AiRUhKELhi/NXgPoaaI8UNHAqdABFxIw0ZmUuWM5nVu4Kg1OPgfj1bYQxAqfvj8+Pj4/Hx+Dj2JtPBdE7lUoCQBQZF6LzOMTVQAqJVyTucUkk3As10aWPc6L05nC0QWUqobpYGhw0K09dzGbxgmuE6+nX1q26NhGpDdGX8q4dhzDntDXlCxJW6G1ZZbYZUbEIYc4TqmC4NuB2f5XztrRAErM74j8f+OB7HvlFY2OyjFwcS5IJ9tZKmpUpN1AxahkEEqfsGEf8taOmcXi0AWSou3+eRBhvzfK3ceqqaA4DruM7X+doDpXLZ0XUyhids6QmwtEZ8kSNT8a6P0UfHvW3bBphQTxGB8wSNeZ1RD+C6f7bn994KoxOVuj8+f//Hvh3bse1tneDzbEASyAUFwW1cpwYECgPKgtcAA6bSHCK4mYbmQ+VumamGeuIRceY/RyKwsM7zPM/zyiOdAAPX2c/n82nEDahsbLMzgqm9Dc768peHIXS8zvN14mN/WBAHcsJqgEEp5vWkw4Db8bl//dxbYYwgafvjx+//ttW9bW2vMTFs9vPpJDWQC65S/bQ1alojP1cApMAFvgw3TR2fJS6jc87Q9HBBdU36C4YpAIx+9rOfWTxZdpJGP59fX1GqBZdNZs/Xz2BxHjAAkGiNUBEgKbSjX8gkpcw5mR2QWPCtqZRFdm1VJNVHEgFIK84j+V3A3KnfTQ6mSD2aCAdwdidPEBy9Uct3SZNpkKvGnUUCcXVHLBbJKk9SC7Hc4EQe70IZMedz+RsmxsNWJr2AE+YW2VZD4lJx0QX5/iXIgsseWD4+H3sVjEU8IKZ0Z5V9mkEwwfoZ2srSWUQubZ8Wc0U+yBmbUK0AxIF0j2sXGXHpXkS4m845R6/hyBVK1Wlq01QDgLiUqY+toM/zm86uQWXzY99aKW89xAS/LWmapR8Xc+pU82QTSHFYdMFFVwogA6qBZXv8eOxNKPxmNyK49vMpNNWDUJiWwXbrC9wagJiF3Zj21g0MJAnIWvA2WDjR+flSre5lRwNG9t2GjjkmISCK2FT72Av5OIl7t6ASeCTTn2/NTPrlYUaupdXqvfdwsLk8HJDcqkV9BQDR4IJF9fh47FXol4cBbI7rSZAs91LIAZKSW//iYQuk8zotdBHRhUVy6A14T1U5eE2q+b7WGbt6YUDBQIBr9M4EEcRWzcz39HCQqQYVlGNvtf43D/+Sptm3zYTAFCyHIRKQVCOExf4N8ACqJSK247FVwUidRyL2cO1C4O6AIrWKB5J4QC3ChAm4q2vYNUD785JE4EmQFPXANUheHs4K/C1mmh4uLCQkjOdVhDDbnOERXqWgj1CKCKAicexbKxlhkrvwPtIAxHU7jsMYfA7waR7AARREELbebIAFeyEiatu2NSHwXx527QxueafaBEv0DpZassgEkoZcGtrg0P56iuQMF7jU5OTkkab0sEhZ0cZ9jVwEEKSWVqnWnMoiJBghiIB8ar/7RpRHevFIif5ypCE9/PE5waZQ2AwHJCEOQAh3vClAECW7XVJLrUUoPLHwxAE2Cc1mwuuH5h2GlHokhJQX4dI05klg/fWV02gpKHUmgeHXkWZRKSa8CB6mc45ShAS5bscuRbLxcz8Kb0gZlVKISinHvtVSZD2Bfw1ay8MfP6bPLgSuCZngNVdKHdtspFChsm3bJsxCgpGFHTJ7uILrvJI/MSY7ACElNGDdYeTi5nZWBu3Pr7JUT6gOtTzSd9ASZRGTtz5velhYKnDdHx+VGcDnLLRQPDTHmDrnwG1DobJte3qYU+vzDlq6PFzb8fFjzKsVApuLdIW+KJUOK79HCa7b4/HgfCbjFgNmAwOzwcxS2t77lAAk4AimdTCRGQIA5lfl0P76qjkfU9761DtorTssYmL3g+aW6mvMZUdp+8dvG6H77ENIOBuMJ4X6uE4MlKDSjn1LFA/8LWgVABAK7deTYfz59f26hhrl4IhgZQ/3RZVb08BvSp5qoNT945zrx9KefeSRdXA48K3j+ou2m6SVrdbaSiv12LcqBG7DdYiwRL/G1XvvX9+XBteDt1YYXXu5qggxzefrSnLl/QFEkmKGtRbhTAzQ5zjj+/tcqF8AqQDAd4dj/Pzz63n2aUCBmZAsi2/tyvdXnpMWCFVHacfnnFldMGX01gF3KzIvqkvgjaXBhNXtrbZWW22PY6tCGOqaVZONPnrvo5/nOYOb1ZKSCaNfiXno32mxh1OCIwOQuThg0icRQ8GGnAzneQ0DaW+DiUAv9HmN76/vNDhx4n9JD2HhOzMTcFvKzKEzSNoxTVe/hB57FXQdt3CBJxrybgQgYoKutj3l8rfajmOvZcWOiIDQm4w45pjBFY2JCVydhREjvL1e59mnRYSTuy32lQPhVkthppRLQsIYY04DrnEbjBQ6Yl4yX89Xyikg8ko5bw/T22CCcLvFxpeHHe0m1m9bZQobamrTTK22ph53c5gAgJhLbduR0OuWAyQG13toNmefY/Q5PNyDWvXMfRRIEsjTzn5dfaqDo7shYgCSABLVWkUIMSzC/SbXALe3wUChPolwXud1nX0aYCq5BS6SLwASSVm6JOF269maBkrzpYQGiFjLOtKr8a9tNwcgQgoMAoI3cHLb9rbtbWutViF0W1JjS8t5jImcERYWJciREMJMs585zYHcHQ0gUsRYqNWUMLRsBunikAngbXBEGESEz9HH6EMNs9m76pr7Dr+PdITBwkDYdJKGXP3mSzAzo9uIpY88N3MHZOGI9C8kcHLTLTnrey0ihSBSYrhfvc81Nii1tsK1kU6d4ToDIcJs1lS8U0dHd0REBwIkDiq1FCFC1zFGn31JYHIiAAoAgE6bS+YrRSUow3IWx9nEfhvMCOEea4xg6igo1bIRj5FuDjXLv27MmRPlwsQIkPKIUmrb7dj3bd+PrTAxEXiM6/U8X69XnzqnDp37sVPh9uDRu1voiGQh9rr+ckdEtBw7IEMEcJGFbbJxnud1ettaI+FKbw876Lh6v/TOVni9QyusLvqdSCm8gpYv6ItFoAgsyEKyyHIwNntPRaFMzKsqJ0P1bbAf+7Hvx7FJqsp6jPP59fz6/r50qupQ/TSswe0oF4SC6/DU2b3qvbCAwhHtrscxNS1YiMC1X8/n82nHw7CANLo97BN8jut86v2m+bq1wJLebTWVhXLSpG6e5ajZ6moCBKbJpqmsN9c0ZLXSxyAWitS/TdxTqa1t+3EclBN8t+v+TJ2qQ7VsBih1L6FKGD4dVvUUC0Obo6Eg4xS8XhNmvEk1r6ehlJwEwI3T4tU++8snmR5EUs3M7Hg8Ho/s1i1FoLstlEAfv0dpgRma1dQMBHmztjVB62RFihe8QR/u7kAkUhuq50ThuhRkAxljjjn7nHe6JLdK8V/m/euDEIoYWEQAmQonZBdiMarpVzfhl8GcP+1OWzJvJiLh9TBu+77tWxVapEuHWHc71lwo7jfWZo6ODQGlBKKUImAjZqvWkPi211abqjYItXGNq09TlL3ss48+Ru1j37eWicRqfeBq7dMi0UFATo6ithTs5rt/lGgoJFjdhL8YLCnc6X/zMCGx4JLlW5NqIVv6kXcSgeE6p+qMO/GYS8kzhG+YBCH6mGIWSBzvtmFCTEvdXMHm9XqdgQAigNqvfvVSrn17e5hoEZ4AFsJuDeBT29QdgANp4YMD4BbiQlpNuL96OJda/NXD+Q7d0u8ihaWwoLnNOXXiEkHAVCUbI26E31wC87DVwlttWwYxdcpo7f/dw9bB5/X8+l6qC+LXebVThI5922ophfmdw6f45S0ICR7qljxLNgcS8QgIj1hc8/tIs/ztDq8aAf7Lkb5BwryWNgSEzT5GJxGR7HBq79d1xuJa0MwpvxIySDuOx5xzTpsTAkjE4tcdTg+3pmsS+HPbNynbvsGrnVWYcd8zK7l3mDitKpXuzE0NQufok1hqIIu42830y6dmkTh+GWz31/c+z3fQkrrt27bt29rxAZYwv/PiUgsgQWqgvV6vWJpUPBfom6WB1OPHj+vsoTbOQJL7wc6RTWQZsQ0Gn+fz5x8fKCDbxwetbtQyOMUP7jADt9AcESIFgdvsV2epll8p+Npwk0NY+G9BCxfS4b95mKXuj+N4PI7QvCiRzfXXizcPIk/0xfV6fruIcGGRMXTMOWbdHHk7Pn9/EUywcRqx1OnxF3IpEEmpDRhtXs+ff0DZUPbP33m1KmDftlZrKciUQYvu+0ZrY4djzlau0hb5HYIs3BbTEQl5Ld74m8E5in4bvHrN0rbHx+fHjw9bkI4ceF6vJ3sAiwOGz369vr/8JkyNlQbrYSD18eMfEnqh99fk0kaOf1L68Q5aLRh8XN8//yW7gmwfv5eSctqxLQ/fPvm1W4gxO1VG4DbH9WrbNEcScV+bA9ZfkO3v+0gTACylexEPTP8KI7hN7pmzYOi0OXVqP5MMPMqSrsJ3tk2c+9QupnDM1yrn+jeLJ69DkdfVpzngDQs9r6EGOV9OFI90uVU2k2KJKYlXd9n3fd+2ra3lLGS8Fqbd0hd+lx8Lec+Yh/7kXwbTGp2sbCmwMoVPAgT32c/z2xOD4+N1XlcfauR3NrZaq1i3re1tq6+XYCjBm1N5q08EgWtnjPOZ+sCJfj+/ZZzDUbaH/vh47NtNJLXZr5Ry6R1f5zCQzcq+79t+bA0XGppZpJRSKhO4jX7KCiLTlrIJYti4CJx+GUxJksoxEwCUbFB7avG8nvvSnY55vs7ex1RasIzFK2Cmuh37sR+ZnQxak+REQyOxlGCw2THsel59OnBumXoVmmcKIsXnR+ogIif78jLXOXq9qI9uKA3qtu/bvu915VmR/etSVdaojpMoOBPJehtM4JP+6mERKQUWnAmJEXy6at6O1mDhe/VKxQZNgQr85WGu2/74eHzsjGEjG7hvD+Pt4TVRPH95+BIGSw8rfn48jr3VApRUlUtNR6+lkZkZCkndt33bt72uWBN6ezgXJHQmvTOf7C8i5oGbF8GCD98ZRlmdyUyKPRTn7LXUUguuLR9x9X71eSuNwHIwsUhrx8fn5+cDw8ZVEDwS8bA4LCxO4Bo6ZF59qCMJgs5O6H5Ox7JF+fF57NtWq6egw7hIR/JHAQGgQGvrnayr3nARLlLqVCYIHQScZb/qmooGktuwuYi0BMtikVLePacl2+G0ssO1RxEJ+hh9jqnyXz0sddsfn7/99hE6rpdQPjx/8zCCT0VCXQrQ6WGICWM6ykbb7WG7PYx6T5nWL9L2DGs1ZTbNbw9behiD7kL9pgAjhvpc5cYyeN3ht3Tbwgha3GOKlZsx5laFlJL82x2Wuh0fn7/948cCwi9pnL/e4VgQezNTc2CksAGuHd0dC9f3HYb7DsNIucXSWmtF7ije9pL1GOl9hy0l0sNwqbwacbakMdx1TRN+eZhFpNx7uHAOV59zWnZtIDHWRQolBd5uGOsvD5e67Y8fv/3jt3E+v6v8X0TpPOCJLfIIIMQwCB3MiIRCyJ8rSscdpWP5qByZhB371ra2t63oVFbFyCitxYjALWzijc/kYAQkRotEff8iTN8D/VuxHJ0Vw23qHRpSYqkU1oUSfa8f5HthGUupddv2LYn3tDSSTN+rCs1zIUB2yoiAICyUUESQRKTsWytMcK8dUQ0EIABSmQbEJRUXtraXyZMQAUxVtZo6pnY3wqJLOjpllhIJbptv0YMbHJzFgAPA0gNiS4M9sjEXhom7yESYSEqVzAx0XM9WCPz6z5+vGbJ9+sfeGF37uK6rJ2Z/6hw6McVr+C2wJ+tLA+to8/pu43//8d2N6iO1HZFlP/bGYJ2YPYBIcqSVMnMREAj30B4IaC1xAp82wlPkP4MWA8BNSljatOBggZyyVymeGICAoYBgS39jccrLrWOgeL0Kgevrjz9fI2T/jOzIz96v87rO6+pzEd6oFAnkwrfQLa8VJmDdxyVFxp9/fl+G9aC70imtVEHrXEr2QAGR0IjvXhouUH02LxfEDUzhZpdiri75m4dvsWOPQEL+VSBn0E5GRXJFcs7PZU33TeMSAp/9+/k8J/D2CR97ZfQsHq/zPHvCw1UlkICklHsJVq5qRAS1iUiI8/l8dqN6iBSpIkWIiRjMueaaAwFAJOPbYEA3M4OlXEpM9KutmqJo8N8NTpiTmwMCI8CC2oSH2gy3qb6oNmAOWb2nYoypXRg+r3Pvo8+QnXDfqmDMfl3JeepLWFsLsAByafd0jW78zmLomI7Rh2GFUkttpVZJEoi6jOaRHiYyY15HGVBtYlqP+ZiO6eZzDKMl1/jfjzTn3EJtaQHyu7E3epjrGLkXMgDMA5HuzrybxpLiae4eIVS5lvTwdV7n+Xq9+r3FxbhYEJc7PYS7uA1dg4dEQWOV1lrbWmucp8NmmdkoEiRiM0+KOyLgnACOmKgeqUUIdNi8ThMugkS/uIe/PHxXkoLMhaXg7WEKo7BxTVgrDXTxte51CWpgs5+l1rW5lYSZBX1Gv87z9XqeVzZ2VaNUB+TSYgWabCkTgo3zOs/zshtNv2UiufF1XVdM65K6cCRI7OapGQUISD07qunhWlsFJbBxvWatAQwo/91gs6WniIQstTSKhQEAGxg6zvfKi3fQuj08fXZhEW6tbSStFcicza4rPbx2JppiVQ+SUlffc00rEMD66/n1/Hp6a601qe3Yj+OxHzt/PzkUtZe57jCGO3tYehgStJiID2Ip29Z8MPjsr2GOVIKY/pvBuopKZUGS2rY36MGnoOu4LmJCJqJMGO9nKUyHrt7/x8cHV9kf9Q7n/TrP1/l6XjfNg8dcHl6fNSeH0H5+/fHnH3/Gx+MDBOvxcXw8Po6PgyvH7GC9JPKO760KRknhychJCL48vO2zENi8Xt2B2QGFAEBOAPA+c3MsEJEIi9bWWqulvD1cam27qjMQAN0Sb6ZmqRP4+Kz3COKuZjUigJBClgjCtJWgJSZnjH5DZQBSNzBSwahuXsuiHOZYqnHbeh+9r8B/fqNnazLmGFdKo67BGFikamXPBAiRWWpte6oe/gGJlzbgGlJrvpS5mkJu9jxCblmte19TfcvuwriqBrfHCJxvg2stqJeVN1rolrxaTUSgcO2XkN+ctZRogAgF2VzaR5RaK3l34lJb8wCWuqsF1l1ivtBeDrmg997mNK7XeV1X79jWt9YXCYr23HH5YACQfwEA5DLSQkW3nBqsl+zXhnEuO3HdHmORAj1J1NdVNaQdRkXxLi4JCe28pJZKVGrx0YsQxKJ/AiIkWkhp5bL3ECMUC/D+MQOBEK0PltrmvdHDgliqwEC7yk3AzRR0qma4653VA7ikwft1XbTtx/F4PD7fHgaAAGCGRdu1m4OEyc0JBK4odU4dvV+9YyikrtRVJ3BzqrutyhL1vqnbTkJ1261flWkxVwA4KGwOApvJg+GEruZQQbhBhCfFc5rUbYzsRjYPkgJEMe2i+/m9dV9S4+7qVxf1dabXB/f9eDwea4XYH/kOJ9opWZ4eYbe0EAAEYABD6gDO1+v1ItelsXrlkaZyjJwtAcHZr+uy6yLHElz2j3m+EiUYGCm5B649fI57afHaXe4hXJiFxa/zumKMU9rep7ojFQ+S0szM5998kjlg+NWv6+q9+zRAKnUbb4OP4zg+Pj74bXCtpRHXSrE0Yu4k8O1h5OyE6FcTijlWS3FcNYCp7uErgSX4+ibven0Ryh5U98d4tRRvufmnGIvukYJYRRZp0GNj2dq2Nf/6+o7h/Vn3q0+1AJYgKU371fvo/Rr3Vrlsc0NE7/3qvfdYHn4bTPu+Px6Pj19HmvY9uHDd3yrgffTeEezXHU6kr2grGNoZ3h4mElqr7pEQCCpZR7u+QTYFLttHT9JW4jMw4V8aysS1llLVCma/xkI22T4+Hx/+L4Hx8v6sx3UfaZRibi+0a76+v1/vkLGGangr+oLafzX4OI7H4y8eJgWqQHVfgRXpPE/GsCRkAgJwTSK9C7mOl6xNReMsrXKttbZF8kIg70/U6wu2hwbV/ePcWx7pZM0wYpgpIFBtrZpFoK9R0A5l//zH778b+3iy92db8DMgzDKB9Yrx+vNfX/f0cu1KJ+pj8TTUU390bNt+poeP4+3hPwGAUYoBl9aWI7EKRdjEe7NOkGTzzMNHP2vuO9TZe+dCZd/3Q5CREDn0+i5o19MfXR2lLkzrL+DEmx+NamYRiLiCZShwe/z2P/6n2fVsFPO197W0dC2RJntRzNfPf/5xK9GsjaDM9yYBXo95a621/bou3o/jcdzPUr60uViq+y1f3XvvCtzQ1FzVNHBpN66vddU3iGgDbLxqk0XupvcMlHy8flaKf/7ruyvWw++OIEQy4FGWWHWYTYtYejFbYX92w7r/uH47Gvs4f94zDvrzOxH+jenOt5mFWPhaSr5L+52XamCf43Hc8odvg9Nd170SOUxtWgjQAHcbcwBRUYsEJN7DWSJCsG6dSXJJ+dZauYXblby/fpKPn398XYb1wLI6Y7j6RpBSdMyuGuAWizbBGN/dsBw/9OPRKMZLlBJiQ9/PcxhQaffsbv0M4ddS8r3tpYQ3TpuPY2+tll+jlvTwHH3e8Mg7UlE4hs7rQpY5bw/fCGREJDAdAQFQj6T80Gr9NCIfL/Lxej2/L6P6KDX31Be6ZTsW1hltYIQhuM7rLIKBz25UDsPtUdnnC/r9Nb++z9xeVEqRKqXKEs/n7ZmRot57SLjUtk3z+VjsB/rbkdYhgquHbTcZgV0xbFwvktKaefyi69xBw2YiOurn56cFlRKAiVQhHy8fr6/Re1essdWt1dZau1+/TA4hYiK4EULY6C+mMDq7Yd2xlNooBmrB5eHrdXaL1LItrdYmq2XNqWzplppvTMxS2lRzvY/0L4PDXXUOiYX8m61tDZgrT0bXeb24tKl/Y+vdzB8b4xpXHy2/+bb5AtoB+vD5kpJla2Xctr1t297WnjaIFauMwU0TLtKZwCbPaViOshMR+bT1ACDS6H1oUMGtba21bSu3PEERQncdUouIMBFLVbUA2/c1oft70GIiX4o7/fHQ4MrNR2J2X9L6yM0vt4vfHh6v1+v5em1dncp2vBUKDXwMhGwxIZdW9v3Y9mPf+M4Z5hJBQDdlAggblIk2BGAtkS3X4R73X6eqakBFEle077nBKNGVYDr6e+Mbl5Rms21vWyt/P9JmOgktE/Dz+k2BG3CLkzF0XK+y7TNn2f8lSoON8+vr59fPXYHq9ph+R2m9ZzypOF7bdhyP/XgcB0OOqpamUu/gOnNBxkz6U73v1Bxj6Jhdl4fX6lgquO2P/diPR12vAxOGzzEuLKUsDxePQLa25QYU/LuHCWCm0sDz1JCmwS1S9Ox61b1PzTfzv3xsnF9//Otff+zBsh3XImuU2qbPcfXr6sfjEYz1eHw8Po7Hx+MhC4UTqT0t5DpWH2GG6xBp24ZStm27XmgxX6++KhN8rwnY8239qCv0M4Rb71eDcq+LKg4ALF5aqa0UfoNLke9Oxy3x4zbnGP2CoeZAnHaP8yVnz4WcGRF1Um40MH+ruc0s2mvVmRvDpOV+x8/j2JIEv5KyeHOoFghjLWWBoOLAUrcddDKCq6bBtP4DV5n1pjel+gwRLsDcasq7u6kjk3KWoakgDkUkFee4tCBp++fG3r9hwvd3D9l/0HEUmC9U/vp6Daf6KJXBxon9nCHbJ9b9H5/JhetDPZBL1Xzf8PF4PD4+Pj8flUMvH89FREfo40q4ciBxaZZ4G2au99b2e1fUr7VuS+bGU0mbUJcaIZ/neV7XdeG97UjH9Tpf58vqljK2dBuMzMJEGMgFSNqYrbH3mC/ovbvstJVaYj7nCed5DefmUjmso5VrgmxYju33z72iz7PPVNQvbZX8x+Px8fHx8XkQwbR+byNBhDtopWxlC2RmEiYut8XEzFKqws1avfmqbnMyI4amuqTKuSym9/Kfcb2e39/f2vat7W3TXx5e+98AuRJXMyUi7+NM4hnvzRER5jwx5pzqVJGZwYYPNg3ey27t4/Mo5OO8bg8vzDWnhz8+t0h9RH9fxftZskCSkssAmFL56G8efgtfk/uC9JsOIgDXUsus1fg8z+s6r47j7eHz9fXz58+5Hfs2tmm/PLyQbYBM4pHcRh9mBvcoWhO+rCnmRrUCEuSYGUkIkeq275V80DrSUm/uz/F4PD4+Pz+2tVh63LBMjLxtYQHEBVDyRiOV+iYNskg1e5OP3TSVQU1n9pTrrEVVc3l6ejg3yuu4nt9//vHH2K/jmNP8bfBNwgJY4y3o5zX7dV2w7xu1fduv89R5vs55Z67hHjbDo5aWvLxSSiGbMWbWZ/fAnB+Pj4/Hx8dne/mY5/l6ZSVAq7ZDxAjkABKlVezJXz0sxTwVOZnIJs0IA7OJAG46Z9WiquVcFvPa6eM6rtfXzz/+sx+r5Prl4Vumb8FOGb9jev/6/gm/GW2y//jxjfqaz58/+1a3rZXaPPGTMw6osh3HUWCBZqeqB3G57ZVjHeliA/T6+vppN/CWV+oPWeC7ZwoHxH/xMBf3WN0vYuUBKcyDGG4255yzmtq8gxbfR9rG9fr+81//eT3WgkB+G7zUOgElyzXB8fL+/a//BKMWsv/4HzxPmK8///f1cTgVrg9Fnzauy0BAts8fP2TJbuQQD1MRlZhFHsfj4+Pj86OMV+j1/a9/2fIwl1tEIhCRk/AIhECUAFpEZBbzgFXzEk8EN4VwzTUDkroe1cp5pcHSx1TVHFl//fzXf57XXAvl3kfa3R3dAzl3QZT5ZO/f//xfQNuHy/7jf8brJ4zXH//n2ZXKTu0YPkH7+ZxlD9k+/vFv0sfiVXlEIGcLllneHuanwLy+//g/3wa3rW1bC5CswFa3BoFI/nKH/a/jPo4wXbxhMmIqWhOAn+Ze3Zb+gevy8LOr5mQIbj2ttyrBjSQkllq344jGaP382r/OEVyPH/XHx7FXISSpHlya/pbP78XTHKggZ9hN/uvUOWYfIxfHRCBzaZvek5XFI7gP/0J9IyCtQjfnwERiiIjgYdOBS3DR1d7BJPpl8lFKrZUZbJxff7Q/v8/hVHY4FqyeYclLJYAcKXKRVkAgl7odj8/YCmj/LvzzNZ23Dx039YGkIbd9+OOxF7TxZI+U9NM5V/W1aq929d7HmLB23RwzEazvHRcRiRMiXkO1pJkKE3EOZZdIroW7AVVu5rDYntLKipm11tbahhzan1+Nf36dGtwe8vE49r3VQm8P31pYiDfbP2FX3TZB60+K12sEtw+c+77vTSiSrGQWrbWCdkF2tBBxYLiGzZ6KOHO0XBM3ce0oOWZu9LsFLCInnFxY3h7O/g0RABEvCSIzNQUAZsC15C8iuNbE29Saaz6DQfvrJ8fz+5zB7aiPVRDTLw+vt+m9Kg1J6naMqbuAdfI5+gjeoHhulyLAW8qTWZh8qIiIkAhTmILb3VMb8+rpYbL756qpQXp4jQ+TqlveHqYUSiLEYA93GxgeNgYumSd0d3ML41pqKbXKWnbSgsH6S0Cv85pB7eHHfzMY1shs2Xsf6amzFtTLx5k7FuWIIoUXCTnBeQk1myGtInOpFUwnhs5+r+fcrt7HGJNyRrpNGUpzjc8SRgPEXEqpd0mEd/MHKP8/RuEaOi4G5lpr44xUrpRr64qkvdtmHNoZtOuYM7gh7GuJyV8MXvjKv3t4mE8ksD6RkYi4vPMdhAwWpfCYiUaSQAEq2+ZzJGY7i90xris1CtnyizSmiRApLf5XD9d2m3sf7dUpADAwRbdxFQau+37cEtiTSqlSS0l6RmvbZLAO1l8R7sFUaNu37e8GL5VV/EWe5VI3jRgWbtPCa621tNJkjXSDpLW9bU3O83ypXacgV6CyHSluYTM3VfbRc2/fGJIZ5x5ElD2sm3gFQCRSant7+D53y9WkNhhC5+U1qGwfH1uukZqMUqpUKZJBa7uIQcP6WlrJjKVtW91arfg2mDDe9qaEPEk1R7rGmDr7HMcDCrfjUW9VAJR2PI7HUb5+gl42niLNgko7ximEocnw7r33/UrVzawpWgAhuCrhMvf/ysO3TuEN1tXZs/cCBlz3x2/H6Ffv0pMHVqRyLbW11jaEUB0AUGsrxK202lpptRYEWBummRmBcNGX3FLhrgEyhob26wyWDbhsTU1Vw9cOzbaXkWT2GZqg4oXqfSMt3x/P/UQeuTLTbb6hFiX3SbV6n2J0SPwgszMHwy0GOFizhySunFuDc/qQyMe2jYELme4aiJIYpryGAADyBZCK04RSmQhC3XAmBiLCbPLqhszRyxULH5siTgxRntfQQGmEPk+O+fr++f3qBlJyYwJAXS0nJBbP1RR5Wku27rd7ktvajXQMW82X1QAn+/76/n51Ddd+FiG4xui9j9FJPYJy3lHbrlYWHs0Ew7WHam1aTC3Lw58AQG0DCpTKAOiBE1QtkAFBZ5LL02CR9/7DMJ0XhsvrHBYkFckHxby+X6/zNQy5RI64YxXzhMSSQ/FFTKlt2zK7TDBWq77m0jZXSHp3zs7XeZ5d3Wa/hMLP+5XnFmv5K0vdzKEkGow1FyjYnE21WqrnpofJgCRIKt8ysmERJMQwhyS6VecYfb29mazowLAp1zUsUFqgz9BLZPQxhoHUSOl3SJmJFMTJ/l1OEUnXOrKtphp4q2oanhrhOQsEAAwE8Az5M1zHiWHa7q9EkqLqQFyqemBJigchEli4Yi43cc/y8AsAKEhqoFQyNc+sAAGZEUeRX0e6MMcvhoBi2JLrCpJw8jEvRHA1MwMuC6YIdY0+kBgAEZmAkIhJty3x7TWHMKUk9td03DziW1rcF5DCfXYMn6OsRTBaUs0isucAyGWMOTi5heEKwbrUx/A+0kylWpA0nG5hY05OsQHCq0qqOuicQ4jjrjDCNUwnY3ZoBN9K+ssp8paDvY80EUNuHc6xKVvi7La91FUCpJyCzZFQtvPlt2j7PY5y7eDar1vd1RqSSILVSqqAjVRMBltCCZghE+LtYZa2a6BUTNp5ylYhF8GWQklmqeCNzkxMwAAWOQCBgABC0cjV59kTEWa8afI5zrvHf+yWdb2wbXuOD+5yh8E0t9301+v7+Xw+bwHfWM0ushk+u8h79aSRlLpkFQJJar0ScRqo4NOmgpmbxzL4JwCUdqRaLBqh67wuq8TIpVJuLA83nUyMECIcgoSLMw230A3h9Hm+zte51dbq0gUACIDci7MgSh7ujEzMIrbvSbuSbLQw+VpiMK7z+f319f2lC38fSa4p1dVmEhlWweUsdS2bLshSbAgzAriHhekYPdzcIP7i4bJ9DAuSBpOT3QUogVwatXIfaZ1ECFGswK3VnbjZIgVJCpCP8/vn1/PYHxYCTGvpFtaS2jiEDhQQzsgkUorv27Hv+37I2nBLlvg7Hf18fv/88+vPqaZTVSH1L1DQAnP0uAQAotS+qScu3Iv7EKIkH1D47NeZG7IiEdPybwAgv30crRAYRCBxKTXNtGygcqmtrOFNFuKrPHOPDAtmMvvrStiQsOTp08QeRBboROgeqd6qpo5cUiG9lEKgmXH13rsG3f9cBMilmBuu276hr5cxMd8IS/3UZkKfFg24qGbJTER+7Pt+7At6+P8CAP799x+PxqFhgFJaUJFC4NqnWpCU7UisP4S74SJGJBYbwqchEfZnrsd11TmIENjcHUjwHaXDdE4dMy83Ct5r0cOmqk3VFEMUBF07WhySg0DJ7dg2UHN101tVllrqiOoI8IBwWJzV5GRIrVvs6/PL4MfHx1E5JmR+hcLEBAY+pwNx3Saz8Nppm62Zt0pvWM5xxut1Xn2a28xRVdYZhFx+2TX6Na6e23eZZKkZkGm/rn5dPWO5VM4lg3Peejfc1iem6rQ5nRaltu7bbXDmCJl6Ws0x9Rwzjm3f9n3f3gbTtm1bexuMXAGSXT9nSl/bXRU6poiN3xUrxMqNxnVe15jqppMgwsrCuVPGYF6bhc7XSVKlisgbnGB6PZ/P7+ertVYbSStz6hxzTkyyLEtttbZam+WuvWGcGTSVVgpD6IDVylgeDiAR1bz/+7bt2y+DUUoRodCIIEZxW0y8mLlbZI3+ASL8XnxBxBQEZAnb1dHHNXIHVkY5yYWCsiZjzATJJf+SrW3IIKVIYRbC0P788+eff359PB4f1GTzxYgdWVncGny1FuvX6L3LZBaSm+uOrjd/AnNbVyrZmpkpbdu27e0vHs5zAzEDc7t1qJq66+3hRSsOAAdwInTkFEBDBM81Bgssoe8OeakFkrzyNw9//fyzTgNx/OVh0Ov585//+c8/fv/dqIbssMSHRm4Sr63cn3md1yVMfJMROXcLqq8eb2r9MCDJwvdxWynsL4PVXM1MgVZ1MsZwsDnWHUZe0ECPCHREQ/FUD6Ew7SkmkwwOtwgznVw9iIO4/vUOX8+vP//VDLg58q1IQqD9+ed//q//9Z+XUT1Cdsouyhw3m7TeghfjbK/ChLSWfnD2gJZiByAhpDI43Zmt89byz9vgyFJdNURQWIQppX76dAsS5GKmZmGphYeA6AWRABDDU+Vh+sqKlMKMGHVNX+tfonS2xjfgqunh9U3o9fzzn//r//hfTvWYIFuZuQ2k78f+2B/7oy1UkfRnguXpdjncOoWZ9Dphiq5ErLAKkrXYMvgfAODni8Hmfd1rETelsNk1AlA4ckcr4AqEeVE4AvAW+blmpGoNQHg4EoKslcRpVZIwZj9f3962PvReoxURruN8ff/841+Pz7Orv40ppbZtf3wcH9udsJZEDYpKqaWVVkIHRNjULJyJEQIQb/UBRJRtkYEIAOR/A0CMMYejNChSE/KT+7DHvIOxaZYntpbk+d3BDiSp6oAz1qnPXiNml6/WWgtnZgAWSKVtRxPQ60l21dwS1ZKF9uOMf/txVPL+5K/8/Jyzj+s8nlvNuXe9ztfz+/vr69w2c0QpuVhk9rkWEnPyemIBZxDREFncAfFtsFuYY2FckSBV4HRcypJD+PfekSXsl+3EAADiYg5Eby2uJWwIRfJX/EX6WE28h3DoRdafN1f0HE71+E35H597QetP+vr6+vr58+fXGL2/ju24neTX+Xp+f339eWWB5DkpHr2PRermlQm+qe4gN57nbfDqnAnSWh9Hi5E+tGBmDnfvzlR1KqpDtoiyDA1kNvOVKK3G5rK3JObPbYatyQMK6OX9Vbdj3/Z930d3aseM+tuPo5L1J3x9ff38+vr62fp1tue25f9tt7jO8/X8/vnzNAeUmqvKdYyr37D+XzM5DADAKMxFPf7iYVwrmks+5veWKR1dgQSotBrLfTrmpIVuz3ceSQC5pGSbqdsvyvH7SNPqFwRQqdsMAvWBxMdxHMeY04ZTPYL3j4+9ovenLw//rNkMadvj8RjqQMvDP88Aktoc8vL167rHiwmZA8JMvTCMyzT7q8HUtm2jUjZeuRGuvPcyktVLTkWz0D46wtryuzwMyGJ2Lw6wWGVFUvNTsxfDDT2R6rurubqb4+PxuPqcCsOpHrKNtm2FbDx1XeGvbGdK3T6vMT2Qr/P1fH5//XwBSW3qEJ70mjPFuWVh8yEV0SAipIymf7vD9HgEFZRWEPJtXR4eJtWRS9tgkRI0SxPNiwoRgYwUde2vs2mafYbwspqvVYggwcG58iX6GDpGH/7xefWp6mROlTZzFha0bv17HWnhFLzZrjENUEp6+OfPJ0ltm94eHtersDCLMd8aYLGoI6W2mUS622DW4LKhbGVB/uMuw71YIJW63zRTIwR3XSs+ltRTEstv/qdbgnVkrautDIvSYoFUDBC96/V6vebZ+1D1qEhYK0OuN/fezzto5dyUZOtqQFLqdb5ez++vP19St32qx02+OFVYWMq92Dl7DeFh1tbKi7eHJbBsjtLawrD5EtQfti2qHUIqLitgmMqayUBEpAQ70uLZpzIgm5nc7xJCtqLzSDciHzCv758/rytXU8EuTao0qblqZ+rrfpbuadY2PVBK3a/zfH5/ff181m1/DHVYv+l1qoiI+VLmIabFozDvU38d6QBI0XH3yAibZbNIrdvmty5OChB7aC6Eg5w8lAVgxNxzxMglUlBtvnf6vQdFsXiMvwRAlmjQwurUWhq4uo3eX2dXB5J241jLWgWYc7lF4xpj9Cveu8QgZY0ZCMLdIQeqbtrO1s52nQwAsgMAt0IYNnrOEREDpe7H0LDj2AqFdnd1N3O9+tq/JKW2Vqv4Cs/ZakDCfvUB9xafe/fT0q9znWP0YcBlM5yfnz8+H8extZLyWQkovK7rNRxkMxBc1O32+Xk0Du194VMYw3Vcrxavq8/kUNdaaiucuVFibMzN5l2k/MVgYQyd/b1+iKRuD3O01lohn5ctdvdqfzuSlLa1bZMxhuroA0uyrvgqgm7DIddYRiTTCzGSadCvobk7QT8/PlKb5U620+DzdWXiJ5UXaaUcx9EY7OpDLQJZKHyO81nivIZaAKcLWiM1U00BMTU3k7SXfhlMtTC6zR455gZCqbsFioqIYCjc+rjTMyKlWuG27+UEBe2vk3ZgrvvWnoSuA2+tHF+qfQh076cdBlxRNv94PB6Px741EpZcLjVHv16vPlNrdM+al0VqbZVDewrwIjGCab9eAtfS5GUptW37hmMOC5sjjTZjKYkX+WVwCnTOy5nZmQGJ6+a4sDLgU0fP6ecEyKZhiq0+jgLWQ/vrm5Eb1ePjEAgdF/raABMRd6F65wgawSTh+DiO4ziOvWTP8iY0ns+xahZf5MSVAYJ6H2oOyEzhc1xCMOY60qW0bd93JEzZyAUMsdQ9p1Q9WAanOjcFi2Rhme/vNn3d/OtKXN8ihBIzl7btx0exTqD99SXSgtvx+QPdemWMsBxT3UtNEcFTPTUIGQmJj33f9/3YBdfj5qajX+drEjOTEBWWIsKFcpCs0XMTbHp4XASoerPka23bccAtYqWWHHzKNir+MhhrYXQl9OJLVloCpewjFV1Mx/l6na/X+ZqrbFtyusdHHS8O7a+fdZvB9fjxj9BxvgTjHhVHtpER6fZwbkAupWxbTpdu9fh1x8+n1saltLrWrErBO4bkQ4YsGK6DwNFdfR3ptu3HI9aYr99bsvFOvxgA5AAAEGEIm7GgvRwkwKW5jeu6Yvq8Jx/P2batOTKk6PnxaK/KYP311T5mcD0+f9dxPetaeX+LpAAiUtylM2OhunhdWZnf4JZYHjaUKtux72voVOG6xhVTex+5iIgJbBK40o0SWXf4ABspobJav0osScpmAJBtjUswLNVXAMkYqEAExBAK7aE9odbfX/NQDypBzKXWtu8tBeHPp10aWLaPH/211cK4FNPcfcGEk9ysOnopKHV/HFtL6EjNxe6Bayx7nV42lHZ8fiRCp9R4Pl8+wUYuhEiVB0W3SQv8sqDz2x6XEITNPlV1miq+0y/4JS+1ZlVgQwczQ2ongF59plhkbXOq2TweuWxzkxgvsEv+9z+/LoWyt4LWn382+Ocf3+cwEKCwcUkyG5AR/fUazu0Dy37s2763JhgaphcufJqvrdjTj1yJaIaYRcBqnzpisp6IiTAMbAFzc1WaEIZ/rf0AQIzk7vR4HMd+dzwwPZxgVQRbmz1Wv8SupW0mpTY1d11x9VE5Bur1RX/88XUqlL3mZm22P//8OrshA4aOjA2Jf485ZnDDWtre2rZtFdHDJr2LddAgrpta7HsrQlmWQYRDYjDeQvTJzXZ3WAt9acWwsPj6fl19mANKSEDw4/HY9237ZfDipArHjT/JwlLYx5iJzy9V1SPsWB8mn3YSxfP5fU2UvQhafwqM76/0MFLYxLCBvxItcOYamaO1JmsVtt/Kl2iBXDYHb3ktjPImOCToHN4avCm44ebEQhII5JMZ3NWfzzP9tNqYchzHse/L4CURl1htNxtj9jFzGW1hzE1zxFKqWQTYkW/nI9yGm7km87bspYD1J+rrfL1ew0AQwYbbEFi5NC0wPa97WUQ1dKpNXOtWyAKlOpDXTNPdlgoZmgdxLj1ABABKyK4riQuIIbgOdNMer+d59qm+6EFU9se+/+1IL/CqiIKN83qdfS2NKWTmFoAixcwA0I+0+KFdZ796H+bmBoVEUHvYWcfofRgyUmjoWMvKIoK2bWu1ba1IEZYilIJGo8Nqq7MFcQGSyKXQkf0TY8ZwQCEJugcMc4aF6cQUnI4wBbA5iqd2lzkhS5EidTu2fdv/65EWKRJg43x+PZ91y44/J3aMsJh7IJAfj8dxHI+j2zWe38/vK2tPqcxofXZeMlYgHuG5/j0cPDz406ny9vEoLMTMDDZDx3mekM0REwsUIKmxmFwGEe5kuYWOOPMTTOkJw7DZMRW0IlzDiZl99N5v4kWtrdZtz17h+0jjLQJoYOP8/vPnV+KJ9lZuVKKVCEDkWAY/8Irx+vnHH9+t1lZbqRxh5nDTJ0DWBvtc8+3uVhyrc3v84x6zwcSwfj6/obTS1GpC6WWtOwoAByfyW+w+V1ysZguYgtvsGEhkHGBhiIjkc8yhM4kXbdve7/1/DVpSykyD//XHvu/bvqvWRDMQFQcAIoHbYOOYrz//8//8+XgcTq3slIsKprKQMAmTh80xx1q0bFapPoLbx++pkIrgjKHjfP6EOqvVrDIyoCeazi3QkRCRRQJZSrkNDp+UHgaibNemdjK4rbWYid099n1hotLgDokUC4QcQ+ucY2aMFoYbPpvPvSAkZLW1IoQQtiB7bX9QHwjuS95JRALAbQFislgbi19zr6vy3q/rus4TgkglbiQpYqx994uin+4GTMtLwvnMpnCSTuGtRoEB4IhIv9hg73ZjOCyVB2wlwbL9TJDZoswjxHqXIQWHApeqI90cgYs+Pj8+Pz4/PqjN3My3xpgci+cqqmrKqoI++/Orkqx2mP/8/v5+nX3SEpPZ1+/9C7ywBh9EhBAGEKnLjFwWzsSg1FzcQaskySM9CDiBZBAzu09JxUuDay2t9jqWgoLWtvSdbqQoJPjqFrJE4tK2/aMP/vj4+Pj4+HzQIhLqm0N275XSqTqZkNHn9fpJcT9U/v18PV9XLt/iUtq2WoVpb+6KuSlOGJ4VWJCglJo7Q1yhlFKYKHezCYv33vug7ERORLdq/xcG11ZbsXNokLR8BlPSWlXV9CYh8PIwsdTt6EPlI3lnD75XTb7hkTfTdc6hPBARXa8Xg3Hai/46z/N19QkOSFLbvlbCr00icGuSJLA5DB0jgJAjDDzczNTXAhnkUlqppcZ5XsLgxhimGKaqVf+rwfnHF24Sish7oc0cY4yFf8fb3jzSQ708Ph6Pj8fjwbZIWoGRMoGZvQjPsURGMWZ/gg2623pX79fVx32kt83UFH65mIDfWgNLYhQWFNAjDZ4uLCxEyCUnVd5qdvzohgtOnbPq3wxutbW61ZhTgwrcMwsImxlYattakLw5AMSlbUM96iM/By+sx73KJPT28GBiRABHnxfYOJdMV6666nNM9uTf7pMmBMLS1UBIPTrOBeQeHs6cWSGkYJtOS6gHApe6H8d+RNo7B2Au1ZKiVecsafAfAECttbq10dDcgoTKG8Oydhe/tsMCpdxHOjciqAdujyP/8BLPC1sdy/mrO0OMEGHoCj4uwbvfr6tiXeuy28YIYLRiMsDSr2YihAXyEmAqpVa0tc80y6X08P74eHxEKtINCQiHgKCqc5bb4D8BgNrW6mhbZwRAlDeZBlI65fv7uZsTt3ir7yKXujkgj5Vq7nLD+dWTYaSJzS/CjAQBjuhTB9K9ZiVL5uQQB3KpbScIZ4Xwu7PLyMxEDBAQrqbBQFL3JuZmU+fQ9TDl4/j544enf2vRtd0C5yyz6t8M7q2NNlrh5JdSJkvv3cU/f34AlqJJ4b6PtAdSmcdx7Mdx7JJOAViLw+bbw2vJikG4RXj4bfCSAFylkNS2QZhNBH/3wW5tIYgEEEwoQNK2o6ipjjn6TMA5IJe6Pz5/+z1y10aVSJllhaJF5yzzLXpAkocAvJbEkYV5WLgHlbo7Sj0ej1bIZ1/DRbShgSSVUhtsst/iHrbExRapI39vJqa6hO9uSQJY+7cAHp9HpZjX8zrP67zOUxPgiLiADpyjx8ol1VAHzdwannvrAQmpUGh/MfifP7+fZ+/TgalEBN7UijclnphTfRuIkWvbqqkaaspKIdf9o2xbK+gjEcMY4FM9SIJysf2AVIPBzBB09j5zpTJyGiw8cR2CWDy6lVoA7Z8pqkR/MTiPdK6tZCEIQKESUgRjotMrcXDmvC56wZgn2rCvnz+/n68+3k/K/aS+DUZihDAw5IpctmMfY85w88ACUvc5qbAIxbw13jAWSzi3gE/wIsJIgEyQ3WWNACCBQEJmFtE721uC1x68YnDd98oxX5bd4PPUG9mYEnJSsiclRESCrjHhufYvBK1UtpBP9HHa9/f39/O8xl15yw32gLfBgCm+TTWQ63Y8Oq89D1Q4uxIIAOAj0bQYgBEByATMGAZhVhbLaCG8L0UkWDs8mKUUZWTMVtTSC1tiWCJFCsWIsQy+Fq4u+TZlEWaLcJHMcxXs+UoPQ0AuzGX0YYNJX6/X63n2iSBt2/atBNyczpswfZ8xVkcp2/HBCK4zPIgzA1dX1dwrlf/CkhDJeO5hWAKRAzF3l89+GQms3Iy5SCmWJQmTJ7XCPBcusxAikM8JfzE4j/8CHFaryMi11cUjsvk6z96nGqb0T60Q7gMi9Eq5hyHI9Xh8PNrScPZ4G2wO5uYmFsi1HR+UXe4IlsyX+rh673P6HV2FOFUIIHFwYUBsAUTrSHctkEvmmVmm6HBh4SIsNz8jZ+ZFCpia2TT9u8EAUGttc1Zz4IJSt93G9NA5xis3bBjl0GFr9z6LsThxo4G0/fO3H5tqwjHeBodqmJvOog5ctuMDzOaF4EElq+fX60k+PPWlAyKqFGHiwm6ZZxhySnEghNnslwYykNTKInMWrb42CZQ1hFRbgmLVr36NOfsa6JyXBi6CSGuzTTXnvG2PiTF99vM6+3WNqcoLwr5183n1q485dM45pgHX4/P3f+yJAh3zl8EYBq5jVA3ksh0fPnPzclDZHsdxHF8/yefps6dsboRXpEApouHhqlq4lFxvlM/3ZdmeKptNKVNtRln7hm1xx63V2lqrbT6/Q2O+vq/rNjgFK6EtpR+obQ2tXdG1v85kJath3FsUh4/r+XwOXcy0XJv7+/945CBQur+P9EIGO8uiHS9eBmaVenw8QntlzKOnt0jRDVAKN8v1PAuL6GY6rSxF5sw7gkJKbVttTftgIUQtdemDDwaf1+u7X/26eh+aQ+VAYlVZosWZhuTILXdrBNwLN2trs69tGAs+56sz8flxLVqNw5othdQ255zz8Y8fj60Q2Nq90XxBjhJSQLmCePX3MBR8Zs2AjDU3hSdwxt88WUQMm7332X3f9kAqkWPiful9mftzfeZqg9/vCN4kvS0nOpeMPjIv81U+lm1rrdVamjoQl9YXUGQuqC4iixXIbsgyeEGs9Pj9x8deOcwhnWtJTNOpFqt3U0qpdWtC6GqKWfmgQCnCix4QN0/2JgnMfp7X6UMDuNgSlTzPmbX2nOP7+5k0JTUN4IhVBFMe+Va3wrk1aabUj5TMw5jLvm2ttlJTByt1qEcfCKkESEQkHgiYO8T3fNiytNs/Px5b4VwsIlLq7eE8P2lwrbXVjQJCAwIWAYdqkVXUxD00zKcFMyF/PU0DqdREGo1+nV1VzWbT8Xx+P7+/n69kkpHcSwYpWxO11coUOih05OaikpgSFrk9fDft+nVeFxN4yogRErsEIsn7SL+p+PVxHHslsKVNqXavrciklTjK8jDcOklFSghzqWVtDM2Z8DtlxgTCf39/qSNJU3tvFe22pKHfRzrfIs4aRpil5nrqWopg2ASzmczXAoisKiq5R64WIC5tjtFf7cmIrqkEQkScvtK3h2/tISxb21rhZbCUdaT/4mHIANFazBE65/BWAxml1LWaeMEc7ocUcdWYX38okrR9OtwePhdO5teRXnU4LRq1lLUjJpXoNWyk/DUJIrGxqN0eriTSVNWu1Dqe4z7SSAzIlhsX08MLakzZZySwyNfcPbE1GvY2OIPWZo5u47pUPYkw5VfQei/IXHW+zX5+//xzsrTtUIs1F3+dlmydXx7OvCsVHxKpuXpL2dMyXIMMZEQyYxNfBpfCnjL8V44dR5ebAQhIyWF4e5gLMxfOhbZIsY60J58SHEJ9IStKqaXWrc2JoeN8zSSBcMntk8vD/vegNfrr+fOPmdogt4fP86lz6mxvg7+ftXpl+P+2dy3JDcMglI+M09z/oJ2p3EiWBV0ATtqmF8iUhVYeC88g89V7tCShpgS1YAAyq3k2CEhMPG1O1XJJk46kszGZHqM576RbzN1rlGuG6EWWQmoJIo7ExXACAJrZdJNmJnfUsmJHPXrbdixyUWQR9sThnN355pZu9eN9l7dr64dajHp8blFfG73WWmvdtmnAilTEIU5lXdgrgTCH2RxzYES0FNdLrcTlcvH6NVBD1GPva3glIjLgAGf52VtCVQPFU9VzajBbRiesCGWskkRTD0//EkcfTNKDAODR55Iv8zz2LLCAxn7oPHauBQA4cmLoFOfff873pgNCLgD0h44vK/8f/OryBZ6pZjRd/4oJAAAAAElFTkSuQmCC " > </div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Be warned, it will succeed sometimes, just not consistently. For comparison, the first attack succeeds with close to 100% (we couldn't make it fail). Actually because we have 10 classes, and if we supposed out-of-distribution probability distribution is uniformly random, it should be something close to 10%, when our initial random image finds a place where the 2 models intersect on the same digit.</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">loss</span><span class="p">(</span><span class="n">outputs</span><span class="p">):</span> <span class="n">entropy1</span> <span class="o">=</span> <span class="n">Categorical</span><span class="p">(</span><span class="n">logits</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="o">.</span><span class="n">entropy</span><span class="p">()</span> <span class="n">entropy2</span> <span class="o">=</span> <span class="n">Categorical</span><span class="p">(</span><span class="n">logits</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span><span class="o">.</span><span class="n">entropy</span><span class="p">()</span> <span class="n">kl_loss1</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;batchmean&#39;</span><span class="p">)</span> <span class="n">kl_loss2</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;batchmean&#39;</span><span class="p">)</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">mse_loss</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="n">loss</span> <span class="o">=</span> <span class="n">entropy1</span> <span class="o">+</span> <span class="n">entropy2</span> <span class="o">+</span> <span class="n">kl_loss1</span> <span class="o">+</span> <span class="n">kl_loss2</span> <span class="o">+</span> <span class="n">distance</span> <span class="k">return</span> <span class="n">loss</span> <span class="k">def</span> <span class="nf">attack_rate</span><span class="p">():</span> <span class="n">attacks</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="n">success</span> <span class="o">=</span> <span class="n">attack</span><span class="p">(</span><span class="n">loss</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">n</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="mf">0.1</span><span class="p">)</span> <span class="k">if</span> <span class="n">success</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;F&quot;</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;.&quot;</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">)</span> <span class="n">attacks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">success</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;&#39;</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Attack success rate </span><span class="si">{</span><span class="nb">sum</span><span class="p">(</span><span class="n">attacks</span><span class="p">)</span><span class="o">/</span><span class="nb">len</span><span class="p">(</span><span class="n">attacks</span><span class="p">)</span> <span class="o">*</span> <span class="mi">100</span><span class="si">:</span><span class="s2">.2f</span><span class="si">}</span><span class="s2">%&quot;</span><span class="p">)</span> <span class="c1"># attack_rate()</span> </pre></div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>The actual attack range seems to stagnate at around 0% (30% if we remove the confidence rate &gt; 80%) with various learning rates and attack steps. There probably are better strategies to attack, this, but the main point is that it became <strong>harder</strong>.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h3 id="Experiment-2">Experiment 2<a class="anchor-link" href="#Experiment-2"> </a></h3><p>Now let's test this on common ood detection for classic datasets. We will add ood detection for the train dataset, just to check that we don't <em>exclude</em> too much of the original dataset. Datasets used will be MNIST, FashionMNIST</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <details class="description"> <summary class="btn btn-sm" data-open="Hide Code" data-close="Show Code"></summary> <p><div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#collapse</span> <span class="kn">from</span> <span class="nn">torchvision.datasets</span> <span class="kn">import</span> <span class="n">MNIST</span><span class="p">,</span> <span class="n">Omniglot</span><span class="p">,</span> <span class="n">FashionMNIST</span> <span class="kn">from</span> <span class="nn">torchvision</span> <span class="kn">import</span> <span class="n">transforms</span> <span class="kn">import</span> <span class="nn">os</span> <span class="k">def</span> <span class="nf">dataset_multi</span><span class="p">(</span><span class="n">dataset_cls</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">transform</span><span class="p">):</span> <span class="c1"># Training settings</span> <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="s1">&#39;PyTorch MNIST Example&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--batch-size&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">64</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;input batch size for training (default: 64)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--test-batch-size&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;input batch size for testing (default: 1000)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--epochs&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">14</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;number of epochs to train (default: 14)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--lr&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">1e-2</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;LR&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;learning rate (default: 1.0)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--gamma&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">float</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mf">0.7</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;M&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Learning rate step gamma (default: 0.7)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--no-cuda&#39;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s1">&#39;store_true&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;disables CUDA training&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--seed&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;S&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;random seed (default: 1)&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--log-interval&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s1">&#39;N&#39;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;how many batches to wait before logging training status&#39;</span><span class="p">)</span> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--save-model&#39;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s1">&#39;store_true&#39;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;For Saving the current Model&#39;</span><span class="p">)</span> <span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span> <span class="n">use_cuda</span> <span class="o">=</span> <span class="ow">not</span> <span class="n">args</span><span class="o">.</span><span class="n">no_cuda</span> <span class="ow">and</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="n">torch</span><span class="o">.</span><span class="n">manual_seed</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">seed</span><span class="p">)</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s2">&quot;cuda&quot;</span> <span class="k">if</span> <span class="n">use_cuda</span> <span class="k">else</span> <span class="s2">&quot;cpu&quot;</span><span class="p">)</span> <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;num_workers&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;pin_memory&#39;</span><span class="p">:</span> <span class="kc">True</span><span class="p">}</span> <span class="k">if</span> <span class="n">use_cuda</span> <span class="k">else</span> <span class="p">{}</span> <span class="n">train_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">dataset_cls</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transform</span><span class="p">),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="n">test_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">dataset_cls</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transform</span><span class="p">),</span> <span class="n">batch_size</span> <span class="o">=</span> <span class="n">args</span><span class="o">.</span><span class="n">test_batch_size</span><span class="p">)</span> <span class="n">optimizer</span> <span class="o">=</span> <span class="n">optim</span><span class="o">.</span><span class="n">Adam</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">parameters</span><span class="p">(),</span> <span class="n">lr</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">lr</span><span class="p">)</span> <span class="n">scheduler</span> <span class="o">=</span> <span class="n">optim</span><span class="o">.</span><span class="n">lr_scheduler</span><span class="o">.</span><span class="n">CyclicLR</span><span class="p">(</span> <span class="n">optimizer</span><span class="p">,</span> <span class="n">base_lr</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">max_lr</span><span class="o">=</span><span class="n">args</span><span class="o">.</span><span class="n">lr</span><span class="p">,</span> <span class="n">cycle_momentum</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">step_size_up</span><span class="o">=</span><span class="mi">200</span> <span class="p">)</span> <span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">epochs</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span> <span class="n">train_multi</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">train_loader</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">,</span> <span class="n">epoch</span><span class="p">)</span> <span class="n">test_multi</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">)</span> <span class="n">scheduler</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="k">if</span> <span class="n">args</span><span class="o">.</span><span class="n">save_model</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">state_dict</span><span class="p">(),</span> <span class="n">filename</span><span class="p">)</span> <span class="k">def</span> <span class="nf">run_datasets</span><span class="p">(</span><span class="n">create_model</span><span class="p">,</span> <span class="n">suffix</span><span class="p">):</span> <span class="n">datasets</span> <span class="o">=</span> <span class="p">[</span><span class="n">MNIST</span><span class="p">,</span> <span class="n">FashionMNIST</span><span class="p">]</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s2">&quot;cuda&quot;</span> <span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s2">&quot;cpu&quot;</span><span class="p">)</span> <span class="k">for</span> <span class="n">dataset_cls</span> <span class="ow">in</span> <span class="n">datasets</span><span class="p">:</span> <span class="n">filename</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">&#39;</span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}{</span><span class="n">suffix</span><span class="si">}</span><span class="s1">.pt&#39;</span> <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span> <span class="k">continue</span> <span class="n">model</span> <span class="o">=</span> <span class="n">create_model</span><span class="p">()</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">transform</span> <span class="o">=</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])</span> <span class="n">dataset_multi</span><span class="p">(</span><span class="n">dataset_cls</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">transform</span><span class="p">)</span> <span class="k">def</span> <span class="nf">create_model</span><span class="p">():</span> <span class="n">model1</span> <span class="o">=</span> <span class="n">Net</span><span class="p">()</span> <span class="n">model2</span> <span class="o">=</span> <span class="n">Net</span><span class="p">()</span> <span class="n">model</span> <span class="o">=</span> <span class="n">MultiNet</span><span class="p">(</span><span class="n">model1</span><span class="p">,</span> <span class="n">model2</span><span class="p">)</span> <span class="k">return</span> <span class="n">model</span> <span class="c1"># run_datasets(create_model, suffix=&#39;&#39;)</span> </pre></div> </div> </div> </div> </p> </details> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">roc_auc_score</span> </pre></div> </div> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">test_datasets</span><span class="p">(</span><span class="n">model_arch</span><span class="p">,</span> <span class="n">suffix</span><span class="p">):</span> <span class="n">datasets</span> <span class="o">=</span> <span class="p">[</span><span class="n">MNIST</span><span class="p">,</span> <span class="n">FashionMNIST</span><span class="p">]</span> <span class="n">batch_size</span> <span class="o">=</span> <span class="mi">500</span> <span class="n">device</span> <span class="o">=</span> <span class="s1">&#39;cuda&#39;</span> <span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s1">&#39;cpu&#39;</span> <span class="k">for</span> <span class="n">dataset_cls</span> <span class="ow">in</span> <span class="n">datasets</span><span class="p">:</span> <span class="n">filename</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">&#39;</span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}{</span><span class="n">suffix</span><span class="si">}</span><span class="s1">.pt&#39;</span> <span class="n">model_arch</span><span class="o">.</span><span class="n">load_state_dict</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">filename</span><span class="p">))</span> <span class="n">model</span> <span class="o">=</span> <span class="n">model_arch</span> <span class="n">test_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">dataset_cls</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">)</span> <span class="n">ref_kl_loss</span> <span class="o">=</span> <span class="n">kl</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Ref loss&quot;</span><span class="p">,</span> <span class="n">ref_kl_loss</span><span class="p">)</span> <span class="n">all_labels</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">all_scores</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">dataset_cls2</span> <span class="ow">in</span> <span class="n">datasets</span><span class="p">:</span> <span class="n">test_loader2</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">dataset_cls2</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span><span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="n">OOD</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">for</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="ow">in</span> <span class="n">test_loader2</span><span class="p">:</span> <span class="n">outputs</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">))</span> <span class="n">kl_loss</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;none&#39;</span><span class="p">),</span> <span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;none&#39;</span><span class="p">))</span> <span class="n">kl_loss</span> <span class="o">=</span> <span class="n">kl_loss</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="n">similar</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="n">normed</span> <span class="o">=</span> <span class="n">kl_loss</span> <span class="o">/</span> <span class="n">ref_kl_loss</span> <span class="n">kl_anomaly</span> <span class="o">=</span> <span class="n">normed</span> <span class="o">&gt;</span> <span class="mi">10</span> <span class="n">non_concordant</span> <span class="o">=</span> <span class="n">similar</span> <span class="o">==</span> <span class="kc">False</span> <span class="n">out_of_distrib</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">kl_anomaly</span> <span class="o">|</span> <span class="n">non_concordant</span><span class="p">)</span> <span class="n">N</span> <span class="o">=</span> <span class="n">normed</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">boolean</span> <span class="o">=</span> <span class="n">dataset_cls2</span> <span class="o">!=</span> <span class="n">dataset_cls</span> <span class="n">all_labels</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span><span class="n">boolean</span><span class="p">]</span> <span class="o">*</span> <span class="n">N</span><span class="p">)</span> <span class="n">all_scores</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">normed</span><span class="o">.</span><span class="n">tolist</span><span class="p">())</span> <span class="n">OOD</span> <span class="o">+=</span> <span class="n">out_of_distrib</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Trained on </span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2"> we detected on </span><span class="si">{</span><span class="n">dataset_cls2</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">OOD</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">test_loader2</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span><span class="si">}</span><span class="s2"> (</span><span class="si">{</span><span class="nb">float</span><span class="p">(</span><span class="n">OOD</span><span class="p">)</span><span class="o">/</span><span class="nb">len</span><span class="p">(</span><span class="n">test_loader2</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span> <span class="o">*</span> <span class="mi">100</span><span class="si">:</span><span class="s2">.2f</span><span class="si">}</span><span class="s2">%) out of distribution&quot;</span><span class="p">)</span> <span class="n">auc</span> <span class="o">=</span> <span class="n">roc_auc_score</span><span class="p">(</span><span class="n">all_labels</span><span class="p">,</span> <span class="n">all_scores</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;AUC for </span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2"> : </span><span class="si">{</span><span class="n">auc</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span> <span class="k">def</span> <span class="nf">exp_2</span><span class="p">():</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s1">&#39;cuda&#39;</span> <span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s1">&#39;cpu&#39;</span><span class="p">)</span> <span class="n">model</span> <span class="o">=</span> <span class="n">MultiNet</span><span class="p">(</span><span class="n">Net</span><span class="p">(),</span> <span class="n">Net</span><span class="p">())</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">test_datasets</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">suffix</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">)</span> <span class="c1"># exp_2()</span> </pre></div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>So we can see that we achieve, with no tuning whatsoever a decent out of distribution detector. We seem to achieve much better AUROC on MNIST, probably because the in-distribution learning seems to be much better (99% test accuracy vs 92% for fastionMNIST). So to False positives for fashionMNIST probably come from this hard to learn in-distribution. Some fine tuning needs to be done to get better results. We also have to keep in mind, that the models to learn this are quite small (2M parameters but only 2 convolution layers) so the lottery hypothesis validity for such a network might be questionned.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h3 id="Experiment-2-bis">Experiment 2 bis<a class="anchor-link" href="#Experiment-2-bis"> </a></h3><p>Same experiment but with fine tuned, larger networks on the same datasets</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">torchvision.models.resnet</span> <span class="kn">import</span> <span class="n">ResNet</span><span class="p">,</span> <span class="n">BasicBlock</span> <span class="k">class</span> <span class="nc">MnistResNet</span><span class="p">(</span><span class="n">ResNet</span><span class="p">):</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="nb">super</span><span class="p">(</span><span class="n">MnistResNet</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">BasicBlock</span><span class="p">,</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span> <span class="n">num_classes</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">conv1</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Conv2d</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="n">kernel_size</span><span class="o">=</span><span class="p">(</span><span class="mi">7</span><span class="p">,</span> <span class="mi">7</span><span class="p">),</span> <span class="n">stride</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">padding</span><span class="o">=</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="n">bias</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span> <span class="k">return</span> <span class="n">F</span><span class="o">.</span><span class="n">log_softmax</span><span class="p">(</span><span class="nb">super</span><span class="p">(</span><span class="n">MnistResNet</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="k">def</span> <span class="nf">run_datasets_res</span><span class="p">():</span> <span class="n">datasets</span> <span class="o">=</span> <span class="p">[</span><span class="n">MNIST</span><span class="p">,</span> <span class="n">FashionMNIST</span><span class="p">]</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s2">&quot;cuda&quot;</span> <span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s2">&quot;cpu&quot;</span><span class="p">)</span> <span class="k">for</span> <span class="n">dataset_cls</span> <span class="ow">in</span> <span class="n">datasets</span><span class="p">:</span> <span class="n">filename</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">&#39;</span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s1">_resnet.pt&#39;</span> <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span> <span class="k">continue</span> <span class="n">multi_res</span> <span class="o">=</span> <span class="n">MultiNet</span><span class="p">(</span><span class="n">MnistResNet</span><span class="p">(),</span> <span class="n">MnistResNet</span><span class="p">())</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">transform</span> <span class="o">=</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Resize</span><span class="p">((</span><span class="mi">224</span><span class="p">,</span> <span class="mi">224</span><span class="p">)),</span><span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))])</span> <span class="n">dataset_multi</span><span class="p">(</span><span class="n">dataset_cls</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">multi_res</span><span class="p">,</span> <span class="n">transform</span><span class="p">)</span> <span class="k">def</span> <span class="nf">test_datasets_bis</span><span class="p">(</span><span class="n">model_arch</span><span class="p">):</span> <span class="n">datasets</span> <span class="o">=</span> <span class="p">[</span><span class="n">MNIST</span><span class="p">,</span> <span class="n">FashionMNIST</span><span class="p">]</span> <span class="n">batch_size</span> <span class="o">=</span> <span class="mi">10</span> <span class="n">device</span> <span class="o">=</span> <span class="s1">&#39;cuda&#39;</span> <span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s1">&#39;cpu&#39;</span> <span class="k">for</span> <span class="n">dataset_cls</span> <span class="ow">in</span> <span class="n">datasets</span><span class="p">:</span> <span class="n">filename</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">&#39;</span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s1">_resnet.pt&#39;</span> <span class="n">model_arch</span><span class="o">.</span><span class="n">load_state_dict</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">filename</span><span class="p">))</span> <span class="n">model</span> <span class="o">=</span> <span class="n">model_arch</span> <span class="n">test_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">dataset_cls</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Resize</span><span class="p">((</span><span class="mi">224</span><span class="p">,</span> <span class="mi">224</span><span class="p">)),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">)</span> <span class="n">ref_kl_loss</span> <span class="o">=</span> <span class="n">kl</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Ref loss&quot;</span><span class="p">,</span> <span class="n">ref_kl_loss</span><span class="p">)</span> <span class="n">all_labels</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">all_scores</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">dataset_cls2</span> <span class="ow">in</span> <span class="n">datasets</span><span class="p">:</span> <span class="n">test_loader2</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">dataset_cls2</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Resize</span><span class="p">((</span><span class="mi">224</span><span class="p">,</span> <span class="mi">224</span><span class="p">)),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span><span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="n">OOD</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">for</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="ow">in</span> <span class="n">test_loader2</span><span class="p">:</span> <span class="n">outputs</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">))</span> <span class="n">kl_loss</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;none&#39;</span><span class="p">),</span> <span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;none&#39;</span><span class="p">))</span> <span class="n">kl_loss</span> <span class="o">=</span> <span class="n">kl_loss</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="n">similar</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="n">normed</span> <span class="o">=</span> <span class="n">kl_loss</span> <span class="o">/</span> <span class="n">ref_kl_loss</span> <span class="n">kl_anomaly</span> <span class="o">=</span> <span class="n">normed</span> <span class="o">&gt;</span> <span class="mi">10</span> <span class="n">non_concordant</span> <span class="o">=</span> <span class="n">similar</span> <span class="o">==</span> <span class="kc">False</span> <span class="n">out_of_distrib</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">kl_anomaly</span> <span class="o">|</span> <span class="n">non_concordant</span><span class="p">)</span> <span class="n">N</span> <span class="o">=</span> <span class="n">normed</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">boolean</span> <span class="o">=</span> <span class="n">dataset_cls2</span> <span class="o">!=</span> <span class="n">dataset_cls</span> <span class="n">all_labels</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span><span class="n">boolean</span><span class="p">]</span> <span class="o">*</span> <span class="n">N</span><span class="p">)</span> <span class="n">all_scores</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">normed</span><span class="o">.</span><span class="n">tolist</span><span class="p">())</span> <span class="n">OOD</span> <span class="o">+=</span> <span class="n">out_of_distrib</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Trained on </span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2"> we detected on </span><span class="si">{</span><span class="n">dataset_cls2</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">OOD</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">test_loader2</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span><span class="si">}</span><span class="s2"> (</span><span class="si">{</span><span class="nb">float</span><span class="p">(</span><span class="n">OOD</span><span class="p">)</span><span class="o">/</span><span class="nb">len</span><span class="p">(</span><span class="n">test_loader2</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span> <span class="o">*</span> <span class="mi">100</span><span class="si">:</span><span class="s2">.2f</span><span class="si">}</span><span class="s2">%) out of distribution&quot;</span><span class="p">)</span> <span class="n">auc</span> <span class="o">=</span> <span class="n">roc_auc_score</span><span class="p">(</span><span class="n">all_labels</span><span class="p">,</span> <span class="n">all_scores</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;AUC for </span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2"> : </span><span class="si">{</span><span class="n">auc</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span> <span class="k">def</span> <span class="nf">exp_2_bis</span><span class="p">():</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s1">&#39;cuda&#39;</span> <span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s1">&#39;cpu&#39;</span><span class="p">)</span> <span class="n">multi_res</span> <span class="o">=</span> <span class="n">MultiNet</span><span class="p">(</span><span class="n">MnistResNet</span><span class="p">(),</span> <span class="n">MnistResNet</span><span class="p">())</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="n">test_datasets_bis</span><span class="p">(</span><span class="n">multi_res</span><span class="p">)</span> <span class="c1"># run_datasets_res()</span> <span class="c1"># exp_2_bis()</span> </pre></div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h3 id="Experiment-3">Experiment 3<a class="anchor-link" href="#Experiment-3"> </a></h3><p>Check that two identical networks (same initalization) actually don't work. It's just a sanity check. We should obtain always kl_div = 0 no matter where we are in the input space.</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">create_same_model</span><span class="p">():</span> <span class="n">model1</span> <span class="o">=</span> <span class="n">Net</span><span class="p">()</span> <span class="n">model</span> <span class="o">=</span> <span class="n">MultiNet</span><span class="p">(</span><span class="n">model1</span><span class="p">,</span> <span class="n">model1</span><span class="p">)</span> <span class="k">return</span> <span class="n">model</span> <span class="k">def</span> <span class="nf">exp_3</span><span class="p">():</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s1">&#39;cuda&#39;</span> <span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s1">&#39;cpu&#39;</span><span class="p">)</span> <span class="n">run_datasets</span><span class="p">(</span><span class="n">create_same_model</span><span class="p">,</span> <span class="n">suffix</span><span class="o">=</span><span class="s1">&#39;_exp3&#39;</span><span class="p">)</span> <span class="n">test_datasets</span><span class="p">(</span><span class="n">create_same_model</span><span class="p">()</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">suffix</span><span class="o">=</span><span class="s1">&#39;_exp3&#39;</span><span class="p">)</span> <span class="c1"># exp_3()</span> </pre></div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h3 id="Experiment-4">Experiment 4<a class="anchor-link" href="#Experiment-4"> </a></h3><p>Run this method with 2, 3, 4, and so on models. We should get exponential improved accuracy, if the random behavious for out-of-distribution for models is correct.</p> </div> </div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="inner_cell"> <div class="input_area"> <div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">create_n_model</span><span class="p">(</span><span class="n">n</span><span class="p">):</span> <span class="n">models</span> <span class="o">=</span> <span class="p">[</span><span class="n">Net</span><span class="p">()</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">)]</span> <span class="n">model</span> <span class="o">=</span> <span class="n">MultiNet</span><span class="p">(</span><span class="o">*</span><span class="n">models</span><span class="p">)</span> <span class="k">return</span> <span class="n">model</span> <span class="k">def</span> <span class="nf">test_datasets_4</span><span class="p">(</span><span class="n">model_arch</span><span class="p">,</span> <span class="n">suffix</span><span class="p">):</span> <span class="n">datasets</span> <span class="o">=</span> <span class="p">[</span><span class="n">MNIST</span><span class="p">,</span> <span class="n">FashionMNIST</span><span class="p">]</span> <span class="n">batch_size</span> <span class="o">=</span> <span class="mi">100</span> <span class="n">device</span> <span class="o">=</span> <span class="s1">&#39;cuda&#39;</span> <span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s1">&#39;cpu&#39;</span> <span class="k">for</span> <span class="n">dataset_cls</span> <span class="ow">in</span> <span class="n">datasets</span><span class="p">:</span> <span class="n">filename</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">&#39;</span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}{</span><span class="n">suffix</span><span class="si">}</span><span class="s1">.pt&#39;</span> <span class="n">model_arch</span><span class="o">.</span><span class="n">load_state_dict</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">filename</span><span class="p">))</span> <span class="n">model</span> <span class="o">=</span> <span class="n">model_arch</span> <span class="n">test_loader</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">dataset_cls</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">)</span> <span class="n">ref_kl_loss</span> <span class="o">=</span> <span class="n">kl</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">,</span> <span class="n">test_loader</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Ref loss&quot;</span><span class="p">,</span> <span class="n">ref_kl_loss</span><span class="p">)</span> <span class="n">all_labels</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">all_scores</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">dataset_cls2</span> <span class="ow">in</span> <span class="n">datasets</span><span class="p">:</span> <span class="n">test_loader2</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">DataLoader</span><span class="p">(</span> <span class="n">dataset_cls2</span><span class="p">(</span><span class="s1">&#39;../data&#39;</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span> <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">((</span><span class="mf">0.1307</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.3081</span><span class="p">,))</span> <span class="p">])),</span><span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="n">OOD</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">for</span> <span class="n">data</span><span class="p">,</span> <span class="n">target</span> <span class="ow">in</span> <span class="n">test_loader2</span><span class="p">:</span> <span class="n">outputs</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">))</span> <span class="n">kl_losses</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">outputs</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">):</span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">outputs</span><span class="p">)):</span> <span class="n">kl_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="n">j</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;none&#39;</span><span class="p">))</span> <span class="n">kl_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">F</span><span class="o">.</span><span class="n">kl_div</span><span class="p">(</span><span class="n">outputs</span><span class="p">[</span><span class="n">j</span><span class="p">],</span> <span class="n">outputs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">exp</span><span class="p">(),</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;none&#39;</span><span class="p">))</span> <span class="n">kl_loss</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">stack</span><span class="p">(</span><span class="n">kl_losses</span><span class="p">,</span> <span class="n">dim</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">dim</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">values</span> <span class="n">kl_loss</span> <span class="o">=</span> <span class="n">kl_loss</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="n">similar</span> <span class="o">=</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="n">outputs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="n">normed</span> <span class="o">=</span> <span class="n">kl_loss</span> <span class="o">/</span> <span class="n">ref_kl_loss</span> <span class="n">kl_anomaly</span> <span class="o">=</span> <span class="n">normed</span> <span class="o">&gt;</span> <span class="mi">10</span> <span class="n">non_concordant</span> <span class="o">=</span> <span class="n">similar</span> <span class="o">==</span> <span class="kc">False</span> <span class="n">out_of_distrib</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">kl_anomaly</span> <span class="o">|</span> <span class="n">non_concordant</span><span class="p">)</span> <span class="n">N</span> <span class="o">=</span> <span class="n">normed</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">boolean</span> <span class="o">=</span> <span class="n">dataset_cls2</span> <span class="o">!=</span> <span class="n">dataset_cls</span> <span class="n">all_labels</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span><span class="n">boolean</span><span class="p">]</span> <span class="o">*</span> <span class="n">N</span><span class="p">)</span> <span class="n">all_scores</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">normed</span><span class="o">.</span><span class="n">tolist</span><span class="p">())</span> <span class="n">OOD</span> <span class="o">+=</span> <span class="n">out_of_distrib</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Trained on </span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2"> we detected on </span><span class="si">{</span><span class="n">dataset_cls2</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">OOD</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">test_loader2</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span><span class="si">}</span><span class="s2"> (</span><span class="si">{</span><span class="nb">float</span><span class="p">(</span><span class="n">OOD</span><span class="p">)</span><span class="o">/</span><span class="nb">len</span><span class="p">(</span><span class="n">test_loader2</span><span class="o">.</span><span class="n">dataset</span><span class="p">)</span> <span class="o">*</span> <span class="mi">100</span><span class="si">:</span><span class="s2">.2f</span><span class="si">}</span><span class="s2">%) out of distribution&quot;</span><span class="p">)</span> <span class="n">auc</span> <span class="o">=</span> <span class="n">roc_auc_score</span><span class="p">(</span><span class="n">all_labels</span><span class="p">,</span> <span class="n">all_scores</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;AUC for </span><span class="si">{</span><span class="n">dataset_cls</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2"> : </span><span class="si">{</span><span class="n">auc</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span> <span class="k">def</span> <span class="nf">exp_4</span><span class="p">():</span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s1">&#39;cuda&#39;</span> <span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s1">&#39;cpu&#39;</span><span class="p">)</span> <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">8</span><span class="p">]:</span> <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;=&#39;</span> <span class="o">*</span> <span class="mi">20</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;N = </span><span class="si">{</span><span class="n">n</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span> <span class="n">run_datasets</span><span class="p">(</span><span class="k">lambda</span><span class="p">:</span> <span class="n">create_n_model</span><span class="p">(</span><span class="n">n</span><span class="p">),</span> <span class="n">suffix</span><span class="o">=</span><span class="sa">f</span><span class="s1">&#39;_exp4_</span><span class="si">{</span><span class="n">n</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span> <span class="n">test_datasets_4</span><span class="p">(</span><span class="n">create_n_model</span><span class="p">(</span><span class="n">n</span><span class="p">)</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">),</span> <span class="n">suffix</span><span class="o">=</span><span class="sa">f</span><span class="s1">&#39;_exp4_</span><span class="si">{</span><span class="n">n</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span> <span class="c1"># exp_4()</span> </pre></div> </div> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>Seems not to be working too great, we ARE improving AUC. Not by a strong margin, it is probably just that we are having a better approximator of our metric by ensembling.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h3 id="Experiment-5">Experiment 5<a class="anchor-link" href="#Experiment-5"> </a></h3><p>Test on a larger output space, like CIFAR-100 and SVHN, to check that part of the limits are actually due to small number of output classes for MNIST/FashionMNIST Other idea is to test on Transformers. Early experiment seems to show that we can use that idea to detect different language within text with just the kl_div used as a distance.</p> <p>Found French book within english books dataset, AND english paragraphs <em>within</em> this french book. Needs some work to clean this experiment</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <ul> <li>Show that small network trained on a single english book enables to detect different languages or different patterns of writing (old english, irish, french, or event dictionnaries)</li> <li>The detection is super fined grained capable of detecting english within a French book.</li> </ul> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>For brevity, we won't include training code. We just trained a simple transformer (6 layers deep) on a english text and checked our metric on some other texts.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p><img src="/images/copied_from_nb/images/self-kl-train-eng.png" alt="title"> <img src="/images/copied_from_nb/images/self-kl-test-eng.png" alt="title"> <img src="/images/copied_from_nb/images/self-kl-test-fr.png" alt="title"></p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h3 id="Experiment-6">Experiment 6<a class="anchor-link" href="#Experiment-6"> </a></h3><p>Need to test with various training schemes, regularization schemes (dropout, batchnorm, l2 penalization) and so on. We should find that the smoother in-distribution our models behave the more this method should work. Hopefully test accuracy <em>should</em> be a good smoothness proxy.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="Limits">Limits<a class="anchor-link" href="#Limits"> </a></h2> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>The pros for this method are that:</p> <ul> <li>It's super simple to implement, and only costs a constant factor in training time.</li> <li>You could also extend this to 3, 4 side models, and it <em>should</em> improve robustness exponentially if the random factors are correct. If we keep this number small, it will still be constant cost factor.</li> <li>It does <em>not</em> require a perturbation model for input data, which in itself is subject to fine-tuning.</li> </ul> <p>The cons is that:</p> <ul> <li>It does not work so well on low dimensional output spaces. </li> <li>It seems other methods have better results than this one.</li> <li>It only works for models that output probability distributions (hard to extend to object detection, generation and other tasks)</li> </ul> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <h2 id="Future-Work">Future Work<a class="anchor-link" href="#Future-Work"> </a></h2> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>There is a lot more experiments necessary to verify that the hypothesis in favor of that approach hold. Try to find ways to implement that in other tasks. How to improve out-of-distribution detection.</p> </div> </div> </div> </div> nicolas 2019-08-06T00:00:00+00:00 2019-08-06T00:00:00+00:00 https://nodata.dev/2019/08/06/model-based-bpe-encodings-3/ <p>In the <a href="/2019/05/16/model-based-bpe-encodings/">first segment</a> we looked into how we could make a BPE based encoding, not only based on frequency in the dataset, but directly on the model probability measure of the next token. In that article I mention that dynamic BPE are costly because they stop being a one time operation but have to be done for every batch because the vocabulary might have changed. In this article I try to completely remove the “static” BPE approach and replace it completely with ML blocks.</p> <blockquote> <h1 id="tl-dr-in-this-article-we-present-an-idea-to-replace-classical-bpe-algorithm-with-a-pure-ml-version-of-it">TL;DR In this article we present an idea to replace classical BPE algorithm with a pure ML version of it.<a class="zola-anchor" href="#tl-dr-in-this-article-we-present-an-idea-to-replace-classical-bpe-algorithm-with-a-pure-ml-version-of-it" aria-label="Anchor link for: tl-dr-in-this-article-we-present-an-idea-to-replace-classical-bpe-algorithm-with-a-pure-ml-version-of-it"><i class="fas fa-link"></i></a> </h1> </blockquote> <h2 id="what-is-the-goal">What is the goal ?<a class="zola-anchor" href="#what-is-the-goal" aria-label="Anchor link for: what-is-the-goal"><i class="fas fa-link"></i></a> </h2> <p>So the goal is to replace BPE algorithm. So it’s go from something like</p> <p>“T|h|e| |c|a|t| |a|t|e| |t|h|e| |a|p|p|l|e|.”</p> <p>To something that has less elements :</p> <p>“The |ca|t |at|e |the| |app|le|.”</p> <p>In one sentence, BPE fuses bytes to form tokens based on frequency in the full dataset. For a more detailed example, look that <a href="/2019/05/16/model-based-bpe-encodings/">the previous article</a>. In this example, you can see there is always a split after a space. That’s a limitation of BPE so actually our target might look different, maybe more like</p> <p>“The cat |at|e |the app|le|.”</p> <p>Here we can notice that “The cat” is a full token and contain 2 actual words. So the goal is to fuse some starting bytes into N tokens (let’s say ~10k) that hopefully capture regularities in our dataset and are at least correlated to frequency in the original dataset like BPE was.</p> <p>Another property we need to have from BPE is that it can encode an arbitrary string of text. It does not matter if it’s not the same language or even if it makes sense, you CAN encode it, that is a very desirable property. It avoids the <a rel="nofollow external" href="https://medium.com/cisco-emerge/creating-semantic-representations-of-out-of-vocabulary-words-for-common-nlp-tasks-842dbdafba18">out-of-vocabulary</a> problem.</p> <h2 id="approach">Approach<a class="zola-anchor" href="#approach" aria-label="Anchor link for: approach"><i class="fas fa-link"></i></a> </h2> <h3 id="tokenization">Tokenization<a class="zola-anchor" href="#tokenization" aria-label="Anchor link for: tokenization"><i class="fas fa-link"></i></a> </h3> <p>So let’s imagine we have a trained transformer like <a rel="nofollow external" href="https://openai.com/blog/better-language-models/">GPT-2</a>. But trained on bytes directly NOT on tokens like the original transformer. Now we can use the idea that when a model is highly confident, it probably means that what it’s about to predict is “in the same token”. Let’s take an example. Try to predict the following Character (as in a single letter) in the next 2 sentences</p> <blockquote> <p>Sentence 1: “Who are yo…”</p> </blockquote> <blockquote> <p>Sentence 2 : “I like …”</p> </blockquote> <p>In the first sentence, normally you would vote with very high confidence for “u”, whereas in the second sentence, you lack a lot of context to be exactly sure on what’s coming next. So “you” would be a token, whereas “like …” can’t be a single token, it has to be at least 2, "like " and "…".</p> <p>Here is a small gif of actual probabilities of the language model on a small sentence</p> <img src="/images/models-2-approach.gif"/> <p>You can see the in the left of the graph the probabilities drop, those are the tokens that try to get predicted but are missing context (because we have very few characters before them. For the right side, you can see the drops in probability are pretty consistent and correspond to word boundaries most often.</p> <h3 id="handling-unknown-tokens">Handling unknown tokens<a class="zola-anchor" href="#handling-unknown-tokens" aria-label="Anchor link for: handling-unknown-tokens"><i class="fas fa-link"></i></a> </h3> <p>Now we know how we are going to “fuse” characters, but we are not done yet. BPE tokens are a discrete SET of identified values from 0 to N (~10k in this experiment). Also BPE can encode an arbitrary new string by using it’s fusion table. So we can’t just run our algorithm on some specific dataset, count all the tokens created and declare that these are the N tokens for eternity. Let’s imagine I feed my algorithm a new sentence, in a different language, French for instance.</p> <p>“J’adore l’Italie.”</p> <p>We can run our “tokenizer” on this, and receive something like this</p> <p>“J|’|ado|re |l’|Ita|lie.”</p> <p>Now “ado” might not be in our original list, so what do we do with it ? Do we declare the token wrong and split it ? That would be odd.</p> <p>A key insight, is to remember that the first step of the discrete “token” once it enters the model (all of them do that, it’s really not specific to transformer or GPT-2) it gets embedded, meaning we go from a number between 1 and N, to a vector in <em>d</em> dimension space (<em>d</em> is between 100 and 1000 generally).</p> <p>For instance token 3 gets mapped to [0.3, -0.15, 1.4, ...] while token 4 gets mapped to [-2.4, -0.014, 0.45, ...]</p> <p>So the idea it to generate directly a token embedding (a vector in <em>d</em>-dimension), not necessarily a discrete value (a number between 0 and vocabulary size).</p> <p>In order to do that we need that all tokens should now be represented in the same way by a <em>d</em> dimension space vector. One way to achieve that is to use an autoencoder.</p> <p><img src="https://upload.wikimedia.org/wikipedia/commons/2/28/Autoencoder_structure.png" alt="" /> or with code</p> <p>The core idea is that when we encounter a new unseen token like "ado" it will still have a representation through the VAE, and will probably be close to a known token like "add". This can help the network overcome odd tokenization or spelling errors.</p> <pre class="giallo z-code"><code data-lang="python"><span class="giallo-l"><span class="z-punctuation z-definition z-comment">#</span><span class="z-comment"># The name is VAE but I didn&#39;t use the internal KL loss in the end as it prevented/slowed down the learning.</span></span> <span class="giallo-l"><span class="z-storage z-type">class</span><span class="z-entity z-name z-type"> VAE</span><span class="z-punctuation">(</span><span class="z-entity z-other z-inherited-class">nn</span><span class="z-punctuation">.</span><span class="z-entity z-other z-inherited-class">Module</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python"> def</span><span class="z-support z-function z-magic z-python"> __init__</span><span class="z-punctuation">(</span><span class="z-variable z-parameter z-function z-language z-special z-self z-python">self</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-support z-type z-python"> super</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">VAE</span><span class="z-punctuation">,</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">)</span><span class="z-punctuation">.</span><span class="z-support z-function z-magic z-python">__init__</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">M</span><span class="z-keyword z-operator"> =</span><span class="z-source"> config</span><span class="z-punctuation">.</span><span class="z-source">CONTEXT_SIZE</span><span class="z-keyword z-operator"> *</span><span class="z-source"> config</span><span class="z-punctuation">.</span><span class="z-source">EMBEDDING_DIM</span></span> <span class="giallo-l"><span class="z-source"> layer</span><span class="z-keyword z-operator"> =</span><span class="z-source"> nn</span><span class="z-punctuation">.</span><span class="z-source">Linear</span></span> <span class="giallo-l"><span class="z-source"> m</span><span class="z-keyword z-operator"> =</span><span class="z-constant z-numeric"> 400</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">fc1</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> layer</span><span class="z-punctuation">(</span><span class="z-variable z-language z-special z-self z-python">self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">M</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> m</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">fc21</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> layer</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">m</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> config</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">EMBEDDING_DIM</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">fc22</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> layer</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">m</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> config</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">EMBEDDING_DIM</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">fc3</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> layer</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">config</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">EMBEDDING_DIM</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> m</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">fc4</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> layer</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">m</span><span class="z-punctuation">,</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">M</span><span class="z-punctuation">)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python"> def</span><span class="z-entity z-name z-function"> encode</span><span class="z-punctuation">(</span><span class="z-variable z-parameter z-function z-language z-special z-self z-python">self</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> x</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> #</span><span class="z-comment"> x is [Batch, Context size, Embedding dim]</span></span> <span class="giallo-l"><span class="z-source"> x</span><span class="z-keyword z-operator"> =</span><span class="z-source"> x</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">view</span><span class="z-punctuation">(</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">1</span><span class="z-punctuation">,</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">M</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-source"> h1</span><span class="z-keyword z-operator"> =</span><span class="z-source"> F</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">relu</span><span class="z-punctuation">(</span><span class="z-variable z-language z-special z-self z-python">self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">fc1</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">x</span><span class="z-punctuation">)</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">fc21</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">h1</span><span class="z-punctuation">)</span><span class="z-punctuation">,</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">fc22</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">h1</span><span class="z-punctuation">)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python"> def</span><span class="z-entity z-name z-function"> reparameterize</span><span class="z-punctuation">(</span><span class="z-variable z-parameter z-function z-language z-special z-self z-python">self</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> mu</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> logvar</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-source"> std</span><span class="z-keyword z-operator"> =</span><span class="z-source"> torch</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">exp</span><span class="z-punctuation">(</span><span class="z-constant z-numeric">0.5</span><span class="z-keyword z-operator"> *</span><span class="z-meta z-function-call z-arguments z-python"> logvar</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-source"> eps</span><span class="z-keyword z-operator"> =</span><span class="z-source"> torch</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">randn_like</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">std</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-source"> mu</span><span class="z-keyword z-operator"> +</span><span class="z-source"> eps</span><span class="z-keyword z-operator"> *</span><span class="z-source"> std</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python"> def</span><span class="z-entity z-name z-function"> decode</span><span class="z-punctuation">(</span><span class="z-variable z-parameter z-function z-language z-special z-self z-python">self</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> z</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-source"> h3</span><span class="z-keyword z-operator"> =</span><span class="z-source"> F</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">relu</span><span class="z-punctuation">(</span><span class="z-variable z-language z-special z-self z-python">self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">fc3</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">z</span><span class="z-punctuation">)</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-source"> torch</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">tanh</span><span class="z-punctuation">(</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">fc4</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">h3</span><span class="z-punctuation">)</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">view</span><span class="z-punctuation">(</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">1</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> config</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">CONTEXT_SIZE</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> config</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">EMBEDDING_DIM</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-punctuation"> )</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python"> def</span><span class="z-entity z-name z-function"> forward</span><span class="z-punctuation">(</span><span class="z-variable z-parameter z-function z-language z-special z-self z-python">self</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> x</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-source"> mu</span><span class="z-punctuation">,</span><span class="z-source"> logvar</span><span class="z-keyword z-operator"> =</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">encode</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">x</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-source"> z</span><span class="z-keyword z-operator"> =</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">reparameterize</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">mu</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> logvar</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-source"> mu</span><span class="z-punctuation">,</span><span class="z-source"> logvar</span><span class="z-punctuation">,</span><span class="z-source"> z</span><span class="z-punctuation">,</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">decode</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">z</span><span class="z-punctuation">)</span></span></code></pre><h3 id="final-network">Final network<a class="zola-anchor" href="#final-network" aria-label="Anchor link for: final-network"><i class="fas fa-link"></i></a> </h3> <img src="/images/model-based-2.png"/> <h2 id="results">Results<a class="zola-anchor" href="#results" aria-label="Anchor link for: results"><i class="fas fa-link"></i></a> </h2> <p>Here is a summary of the values of the tokenization we got.</p> <table><thead><tr><th></th><th>Raw</th><th>BPE</th><th>Model based</th></tr></thead><tbody> <tr><td>Vocabulary size</td><td>256</td><td>10000</td><td>26262</td></tr> <tr><td>#Tokens</td><td>387k</td><td>90k</td><td>92k</td></tr> <tr><td>Avg token length</td><td>1</td><td>3.3</td><td>6.65</td></tr> </tbody></table> <p>Here is a excerpt of the kind of tokenization we created</p> <pre><i>|He w|as on|e of| the |most |n|oticea|ble member|s of the| Reform| Club|, |th|ough| he| s|eemed |always |to |avoid |att|racting at|tention|; an en|ig|mat|i|cal |p|erson|age|,| |ab|out whom l|ittle| was |known|, |e|xc|ept that |he| w|as |a |poli|shed m|an| o|f |th|e |wo|rld|. |Pe|ople sa|id| that h|e |re|sembl|ed| |Byron|--at least| t|hat |his hea|d w|as |Byronic|; |but| he was |a |b|earde|d, tranquil| Byron|, who| |might live| on a |thousand year|s |w|ithout g|r|owing o|ld|.| |Certainly| an| English|man|, it |was |m|ore |doubt|ful w|h|ether |Phileas Fogg| w|as |a |London|er|.</i></pre> <p><a href="/txt/80day_tokenized_exp2.txt">Full text</a></p> <p>This version has been done with epsilon=0.0015.</p> <p>As you can see, "Phileas Fogg" is already a token in this situation, which is a multi-word token not achievable by regular BPE. You can also see, a lot of words contain only single bytes tokens which is why this method compresses LESS than regular BPE at the same vocabulary size. Another note is that classical words like "was" is already a token (in the last sentence) but it's not always the case, this token is context dependent now !</p> <h2 id="vae">VAE<a class="zola-anchor" href="#vae" aria-label="Anchor link for: vae"><i class="fas fa-link"></i></a> </h2> <p>After the VAE step, the reconstruction is not perfect yet perfectly legible.</p> <pre><i>|He w|as on|e of| the |most |n|oticea|ihe member|s of the| reform| Club|, |th|ough| he| s|eemed |always |to |asoid |att|nacting at|tention|, an en|ig|mat|i|cal |p|erson|age|,| |ab| it whom l|ittle| was | nown|, |e|xc| pt that |he| w|as |a |poli|shed m|an| o|f |th|e |wo|rld|. |Pe|ople sa|id| that h|e |re|sembl|ed| |pyron| cat least| t|hat |has hea|d w|as |blronic|; |but| he was |a |b|earde|in tranquil| pyron| who| |eight live| on a |dar and year|s |w|ithout g|r|owing o|ld|.| |rertainly| an| English|man|, it |was |m|ore |doubt|ful w|h|ether |Phileas Fogg| w|as |a |London|er|.</i></pre> <p><a href="/txt/80day_reconstructed2.txt">Full text</a></p> <p>Most of the errors tend to lie in the first characters of <em>long tokens</em>.That's because, I'm forced to padd the input of the VAE and to mask that padding. In practice that means that the first characters of long tokens get updated less that the others so necessarily they contain more errors. <a href="https://nodata.dev/2019/08/06/model-based-bpe-encodings-3/#notes">More information</a>.</p> <h2 id="upper-level">Upper level<a class="zola-anchor" href="#upper-level" aria-label="Anchor link for: upper-level"><i class="fas fa-link"></i></a> </h2> <p>In order to complete the experiment, we need to check that the original language model done directly at BPE level can be done with this new model-based BPE encoding.</p> <p>It's pretty slow to train that upper level because we need to flow the gradients all the way through the VAE decoder, and the lower layer decoding step, in order to get the <strong>character level loss</strong> (softmax + nll_loss) to properly train something. That's a limit of the current approach.</p> <p>If we randomly split the text into train&amp;validation, we can learn almost perfectly (97% top-1 character level accuracy) the language model on top of that Model based BPE.</p> <img src="/images/models-2-overfit.png"/> <p>However this can be considered <strong>overfitting</strong> because even though a specific input was never seen in the valid set, a very close one <em>was</em>.</p> <p>If instead we try to compare with a fixed split, where the last part of the book is considered the valid set, then we get much lower result.</p> <p>We could achieve 25% exact character matching, and ~77% top-10 character matching on the valid set, which is the end of the book ! The same results happen with BPE, even worse ! we can't get past 13% top-1 and 25% top-10 on the regular BPE. That's understandable because the dataset is very small and the last part of the book is different so it's very hard to infer it from just the beginning and no other text.</p> <p>Another note, is that model based BPE are not tokenizing deterministicly, there is some variance to it, depending on the context of a particular word. This actually seems to be a good property (See <a rel="nofollow external" href="https://arxiv.org/abs/1804.10959">this</a>) and might explain away the better performance of model based BPE over regular BPE. Keep in mind it's 25% of the <strong>characters</strong> that are correct. If we looked at a discrete view of <strong>tokens</strong> we probably would have a much higher prediction rate (it's left for future work for now).</p> <p>Here is a picture from the tensorboard values, P_1 is probability that the character predicted is the correct one, P_10 is that it is in the top-10 values.</p> <img src="/images/models-2-upper.png"/> <p>The overfitting starts happening around the ~1M steps mark.</p> <h3 id="notes">Notes<a class="zola-anchor" href="#notes" aria-label="Anchor link for: notes"><i class="fas fa-link"></i></a> </h3> <ul> <li>In the experiment we learned model by model, freezing the lower model before training something on top. It's because the batching of different layers occur differently. Learning the whole thing end-to-end is probably going to need some thought. The batching is easy for the lower level, every batch needs a tensor of shape CONTEXT_SIZE (=64) of [0-255] ints. For the VAE, we need to have a variable length (depending on the length token) times EMBEDDING_DIM (=128). The upper level needs only tensors of size CONTEXT_SIZE * EMBEDDING_DIM yet if we want to try and end-to-end training, we have <strong>no idea</strong> how many bytes we need to generate 1 correct tensor in the upper layer. We know it's no more than CONTEXT_SIZE² but that would be prohibitive to use that value.</li> <li>The loss NEEDS to always be the byte-level nll loss. At first I thought a simple MSE loss in the embedding space could be enough to learn the proper models. It seems to not be the case. I could only achieve meaningful results by always referring to the original strings and calculating the NLL Loss. When using this loss, the MSE actually <em>increases</em>. This leads me to think that encoding/decoding + softmax are highly anisotropic operators. Looking at the singular values of the embedding matrix, we can see that the highest one is 7.35, the lowest one 0.12, so there are 2 orders of magnitude between the 2. This anisotropy means that the MSE loss which considers all dimensions of the embeddding equal is actually couting way too much some irrelevant dimensions. It would be much faster and simpler if we could train directly on MSE (it would enable us to train without running all the decoding steps to generate the loss). So we need to add some spectral loss on the embedding on the lower language model to test that hypothesis.</li> <li>The tokens have variable lengths. In order to fix this, we have to padd all sequences during learning. Because we padd, we have to mask the padding during training for both VAE and upper LM. Keeping track of this is pretty nifty and it means gradients on rarely used places will rarely get updated. So we will almost surely miss some letters in our tokens. Either at the front or the end of the token depending on how we padd the tokens.</li> </ul> <h2 id="future-work"><strong>Future work</strong><a class="zola-anchor" href="#future-work" aria-label="Anchor link for: future-work"><i class="fas fa-link"></i></a> </h2> <ul> <li>Actually testing discretizing the tokens to compare with the regular BPE. In that direction, also comparing with a randomized tokenizer as used in <a rel="nofollow external" href="https://github.com/google/sentencepiece">SentencePiece</a> to make sure the results are actually comparable and are indeed linked to tokenization variance.</li> <li>The masking problem really seems to be a current limit of the model. Finding a workaround would be really valuable.</li> <li>The fact that the NLL loss is required slows down upper layers. It would be awesome if we could smooth out the encoding/decoding matrix so that L2 directly for VAE and the upper layer works. It probably goes against regular language model embedding so not sure it's doable.</li> <li>Making the epsilon based tokenization directly after the embedding layer. This would help <em>stack</em> those levels hopefully learning higher and higer representations of text leading the sentence embedding and so on.</li> <li>On the same idea, another direction would be to do actual discrete tokenization to allow for the models to stack.</li> </ul> nicolas 2019-06-06T00:00:00+00:00 2019-06-06T00:00:00+00:00 https://nodata.dev/2019/06/06/model-based-bpe-encodings-2/ <p>In the <a href="/2019/05/16/model-based-bpe-encodings/">first segment</a> we looked into how we could make a BPE based encoding, not only based on frequency in the dataset, but directly on the model probability measure of the next token. In that article I mention that dynamic BPE are costly because they stop being a one time operation but have to be done for every batch because the vocabulary might have changed. In this article I try to completely remove the “static” BPE approach and replace it completely with ML blocks.</p> <blockquote> <h1 id="tl-dr-in-this-article-we-present-an-idea-to-replace-classical-bpe-algorithm-with-a-pure-ml-version-of-it">TL;DR In this article we present an idea to replace classical BPE algorithm with a pure ML version of it.<a class="zola-anchor" href="#tl-dr-in-this-article-we-present-an-idea-to-replace-classical-bpe-algorithm-with-a-pure-ml-version-of-it" aria-label="Anchor link for: tl-dr-in-this-article-we-present-an-idea-to-replace-classical-bpe-algorithm-with-a-pure-ml-version-of-it"><i class="fas fa-link"></i></a> </h1> </blockquote> <h2 id="what-is-the-goal">What is the goal ?<a class="zola-anchor" href="#what-is-the-goal" aria-label="Anchor link for: what-is-the-goal"><i class="fas fa-link"></i></a> </h2> <p>So the goal is to replace BPE algorithm. So it’s go from something like</p> <p>“T|h|e| |c|a|t| |a|t|e| |t|h|e| |a|p|p|l|e|.”</p> <p>To something that has less elements :</p> <p>“The |ca|t |at|e |the| |app|le|.”</p> <p>In one sentence, BPE fuses bytes to form tokens based on frequency in the full dataset. For a more detailed example, look that <a href="/2019/05/16/model-based-bpe-encodings/">the previous article</a>. In this example, you can see there is always a split after a space. That’s a limitation of BPE so actually our target might look different, maybe more like</p> <p>“The cat |at|e |the app|le|.”</p> <p>Here we can notice that “The cat” is a full token and contain 2 actual words. So the goal is to fuse some starting bytes into N tokens (let’s say ~10k) that hopefully capture regularities in our dataset and are at least correlated to frequency in the original dataset like BPE was.</p> <p>Another property we need to have from BPE is that it can encode an arbitrary string of text. It does not matter if it’s not the same language or even if it makes sense, you CAN encode it, that is a very desirable property. It avoids the <a rel="nofollow external" href="https://medium.com/cisco-emerge/creating-semantic-representations-of-out-of-vocabulary-words-for-common-nlp-tasks-842dbdafba18">out-of-vocabulary</a> problem.</p> <h2 id="approach">Approach<a class="zola-anchor" href="#approach" aria-label="Anchor link for: approach"><i class="fas fa-link"></i></a> </h2> <h3 id="tokenization">Tokenization<a class="zola-anchor" href="#tokenization" aria-label="Anchor link for: tokenization"><i class="fas fa-link"></i></a> </h3> <p>So let’s imagine we have a trained transformer like <a rel="nofollow external" href="https://openai.com/blog/better-language-models/">GPT-2</a>. But trained on bytes directly NOT on tokens like the original transformer. Now we can use the idea that when a model is highly confident, it probably means that what it’s about to predict is “in the same token”. Let’s take an example. Try to predict the following Character (as in a single letter) in the next 2 sentences</p> <blockquote> <p>Sentence 1: “Who are yo…”</p> </blockquote> <blockquote> <p>Sentence 2 : “I like …”</p> </blockquote> <p>In the first sentence, normally you would vote with very high confidence for “u”, whereas in the second sentence, you lack a lot of context to be exactly sure on what’s coming next. So “you” would be a token, whereas “like …” can’t be a single token, it has to be at least 2, "like " and "…".</p> <p>Here is a small gif of actual probabilities of the language model on a small sentence</p> <img src="/images/models-2-approach.gif"/> <p>You can see the in the left of the graph the probabilities drop, those are the tokens that try to get predicted but are missing context (because we have very few characters before them. For the right side, you can see the drops in probability are pretty consistent and correspond to word boundaries most often.</p> <h3 id="handling-unknown-tokens">Handling unknown tokens<a class="zola-anchor" href="#handling-unknown-tokens" aria-label="Anchor link for: handling-unknown-tokens"><i class="fas fa-link"></i></a> </h3> <p>Now we know how we are going to “fuse” characters, but we are not done yet. BPE tokens are a discrete SET of identified values from 0 to N (~10k in this experiment). Also BPE can encode an arbitrary new string by using it’s fusion table. So we can’t just run our algorithm on some specific dataset, count all the tokens created and declare that these are the N tokens for eternity. Let’s imagine I feed my algorithm a new sentence, in a different language, French for instance.</p> <p>“J’adore l’Italie.”</p> <p>We can run our “tokenizer” on this, and receive something like this</p> <p>“J|’|ado|re |l’|Ita|lie.”</p> <p>Now “ado” might not be in our original list, so what do we do with it ? Do we declare the token wrong and split it ? That would be odd.</p> <p>A key insight, is to remember that the first step of the discrete “token” once it enters the model (all of them do that, it’s really not specific to transformer or GPT-2) it gets embedded, meaning we go from a number between 1 and N, to a vector in <em>d</em> dimension space (<em>d</em> is between 100 and 1000 generally).</p> <p>For instance token 3 gets mapped to [0.3, -0.15, 1.4, ...] while token 4 gets mapped to [-2.4, -0.014, 0.45, ...]</p> <p>So the idea it to generate directly a token embedding (a vector in <em>d</em>-dimension), not necessarily a discrete value (a number between 0 and vocabulary size).</p> <p>In order to do that we need that all tokens should now be represented in the same way by a <em>d</em> dimension space vector. One way to achieve that is to use an autoencoder.</p> <p><img src="https://upload.wikimedia.org/wikipedia/commons/2/28/Autoencoder_structure.png" alt="" /> or with code</p> <p>The core idea is that when we encounter a new unseen token like "ado" it will still have a representation through the VAE, and will probably be close to a known token like "add". This can help the network overcome odd tokenization or spelling errors.</p> <pre class="giallo z-code"><code data-lang="python"><span class="giallo-l"><span class="z-punctuation z-definition z-comment">#</span><span class="z-comment"># The name is VAE but I didn&#39;t use the internal KL loss in the end as it prevented/slowed down the learning.</span></span> <span class="giallo-l"><span class="z-storage z-type">class</span><span class="z-entity z-name z-type"> VAE</span><span class="z-punctuation">(</span><span class="z-entity z-other z-inherited-class">nn</span><span class="z-punctuation">.</span><span class="z-entity z-other z-inherited-class">Module</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python"> def</span><span class="z-support z-function z-magic z-python"> __init__</span><span class="z-punctuation">(</span><span class="z-variable z-parameter z-function z-language z-special z-self z-python">self</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-support z-type z-python"> super</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">VAE</span><span class="z-punctuation">,</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">)</span><span class="z-punctuation">.</span><span class="z-support z-function z-magic z-python">__init__</span><span class="z-punctuation">(</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">M</span><span class="z-keyword z-operator"> =</span><span class="z-source"> config</span><span class="z-punctuation">.</span><span class="z-source">CONTEXT_SIZE</span><span class="z-keyword z-operator"> *</span><span class="z-source"> config</span><span class="z-punctuation">.</span><span class="z-source">EMBEDDING_DIM</span></span> <span class="giallo-l"><span class="z-source"> layer</span><span class="z-keyword z-operator"> =</span><span class="z-source"> nn</span><span class="z-punctuation">.</span><span class="z-source">Linear</span></span> <span class="giallo-l"><span class="z-source"> m</span><span class="z-keyword z-operator"> =</span><span class="z-constant z-numeric"> 400</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">fc1</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> layer</span><span class="z-punctuation">(</span><span class="z-variable z-language z-special z-self z-python">self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">M</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> m</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">fc21</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> layer</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">m</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> config</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">EMBEDDING_DIM</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">fc22</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> layer</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">m</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> config</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">EMBEDDING_DIM</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">fc3</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> layer</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">config</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">EMBEDDING_DIM</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> m</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-source">fc4</span><span class="z-keyword z-operator"> =</span><span class="z-meta z-function-call z-python"> layer</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">m</span><span class="z-punctuation">,</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">M</span><span class="z-punctuation">)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python"> def</span><span class="z-entity z-name z-function"> encode</span><span class="z-punctuation">(</span><span class="z-variable z-parameter z-function z-language z-special z-self z-python">self</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> x</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> #</span><span class="z-comment"> x is [Batch, Context size, Embedding dim]</span></span> <span class="giallo-l"><span class="z-source"> x</span><span class="z-keyword z-operator"> =</span><span class="z-source"> x</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">view</span><span class="z-punctuation">(</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">1</span><span class="z-punctuation">,</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">M</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-source"> h1</span><span class="z-keyword z-operator"> =</span><span class="z-source"> F</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">relu</span><span class="z-punctuation">(</span><span class="z-variable z-language z-special z-self z-python">self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">fc1</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">x</span><span class="z-punctuation">)</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">fc21</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">h1</span><span class="z-punctuation">)</span><span class="z-punctuation">,</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">fc22</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">h1</span><span class="z-punctuation">)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python"> def</span><span class="z-entity z-name z-function"> reparameterize</span><span class="z-punctuation">(</span><span class="z-variable z-parameter z-function z-language z-special z-self z-python">self</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> mu</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> logvar</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-source"> std</span><span class="z-keyword z-operator"> =</span><span class="z-source"> torch</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">exp</span><span class="z-punctuation">(</span><span class="z-constant z-numeric">0.5</span><span class="z-keyword z-operator"> *</span><span class="z-meta z-function-call z-arguments z-python"> logvar</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-source"> eps</span><span class="z-keyword z-operator"> =</span><span class="z-source"> torch</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">randn_like</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">std</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-source"> mu</span><span class="z-keyword z-operator"> +</span><span class="z-source"> eps</span><span class="z-keyword z-operator"> *</span><span class="z-source"> std</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python"> def</span><span class="z-entity z-name z-function"> decode</span><span class="z-punctuation">(</span><span class="z-variable z-parameter z-function z-language z-special z-self z-python">self</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> z</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-source"> h3</span><span class="z-keyword z-operator"> =</span><span class="z-source"> F</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">relu</span><span class="z-punctuation">(</span><span class="z-variable z-language z-special z-self z-python">self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">fc3</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">z</span><span class="z-punctuation">)</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-source"> torch</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">tanh</span><span class="z-punctuation">(</span></span> <span class="giallo-l"><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">fc4</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">h3</span><span class="z-punctuation">)</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">view</span><span class="z-punctuation">(</span><span class="z-keyword z-operator">-</span><span class="z-constant z-numeric">1</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> config</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">CONTEXT_SIZE</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> config</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-arguments z-python">EMBEDDING_DIM</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-punctuation"> )</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-function z-python"> def</span><span class="z-entity z-name z-function"> forward</span><span class="z-punctuation">(</span><span class="z-variable z-parameter z-function z-language z-special z-self z-python">self</span><span class="z-punctuation">,</span><span class="z-variable z-parameter"> x</span><span class="z-punctuation">)</span><span class="z-punctuation">:</span></span> <span class="giallo-l"><span class="z-source"> mu</span><span class="z-punctuation">,</span><span class="z-source"> logvar</span><span class="z-keyword z-operator"> =</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">encode</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">x</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-source"> z</span><span class="z-keyword z-operator"> =</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">reparameterize</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">mu</span><span class="z-punctuation">,</span><span class="z-meta z-function-call z-arguments z-python"> logvar</span><span class="z-punctuation">)</span></span> <span class="giallo-l"><span class="z-keyword z-control z-flow z-python"> return</span><span class="z-source"> mu</span><span class="z-punctuation">,</span><span class="z-source"> logvar</span><span class="z-punctuation">,</span><span class="z-source"> z</span><span class="z-punctuation">,</span><span class="z-variable z-language z-special z-self z-python"> self</span><span class="z-punctuation">.</span><span class="z-meta z-function-call z-python">decode</span><span class="z-punctuation">(</span><span class="z-meta z-function-call z-arguments z-python">z</span><span class="z-punctuation">)</span></span></code></pre><h3 id="final-network">Final network<a class="zola-anchor" href="#final-network" aria-label="Anchor link for: final-network"><i class="fas fa-link"></i></a> </h3> <img src="/images/model-based-2.png"/> <h2 id="results">Results<a class="zola-anchor" href="#results" aria-label="Anchor link for: results"><i class="fas fa-link"></i></a> </h2> <p>Here is a summary of the values of the tokenization we got.</p> <table><thead><tr><th></th><th>Raw</th><th>BPE</th><th>Model based</th></tr></thead><tbody> <tr><td>Vocabulary size</td><td>256</td><td>10000</td><td>26262</td></tr> <tr><td>#Tokens</td><td>387k</td><td>90k</td><td>92k</td></tr> <tr><td>Avg token length</td><td>1</td><td>3.3</td><td>6.65</td></tr> </tbody></table> <p>Here is a excerpt of the kind of tokenization we created</p> <pre><i>|He w|as on|e of| the |most |n|oticea|ble member|s of the| Reform| Club|, |th|ough| he| s|eemed |always |to |avoid |att|racting at|tention|; an en|ig|mat|i|cal |p|erson|age|,| |ab|out whom l|ittle| was |known|, |e|xc|ept that |he| w|as |a |poli|shed m|an| o|f |th|e |wo|rld|. |Pe|ople sa|id| that h|e |re|sembl|ed| |Byron|--at least| t|hat |his hea|d w|as |Byronic|; |but| he was |a |b|earde|d, tranquil| Byron|, who| |might live| on a |thousand year|s |w|ithout g|r|owing o|ld|.| |Certainly| an| English|man|, it |was |m|ore |doubt|ful w|h|ether |Phileas Fogg| w|as |a |London|er|.</i></pre> <p><a href="/txt/80day_tokenized_exp2.txt">Full text</a></p> <p>This version has been done with epsilon=0.0015.</p> <p>As you can see, "Phileas Fogg" is already a token in this situation, which is a multi-word token not achievable by regular BPE. You can also see, a lot of words contain only single bytes tokens which is why this method compresses LESS than regular BPE at the same vocabulary size. Another note is that classical words like "was" is already a token (in the last sentence) but it's not always the case, this token is context dependent now !</p> <h2 id="vae">VAE<a class="zola-anchor" href="#vae" aria-label="Anchor link for: vae"><i class="fas fa-link"></i></a> </h2> <p>After the VAE step, the reconstruction is not perfect yet perfectly legible.</p> <pre><i>|He w|as on|e of| the |most |n|oticea|ihe member|s of the| reform| Club|, |th|ough| he| s|eemed |always |to |asoid |att|nacting at|tention|, an en|ig|mat|i|cal |p|erson|age|,| |ab| it whom l|ittle| was | nown|, |e|xc| pt that |he| w|as |a |poli|shed m|an| o|f |th|e |wo|rld|. |Pe|ople sa|id| that h|e |re|sembl|ed| |pyron| cat least| t|hat |has hea|d w|as |blronic|; |but| he was |a |b|earde|in tranquil| pyron| who| |eight live| on a |dar and year|s |w|ithout g|r|owing o|ld|.| |rertainly| an| English|man|, it |was |m|ore |doubt|ful w|h|ether |Phileas Fogg| w|as |a |London|er|.</i></pre> <p><a href="/txt/80day_reconstructed2.txt">Full text</a></p> <p>Most of the errors tend to lie in the first characters of <em>long tokens</em>.That's because, I'm forced to padd the input of the VAE and to mask that padding. In practice that means that the first characters of long tokens get updated less that the others so necessarily they contain more errors. <a href="https://nodata.dev/2019/06/06/model-based-bpe-encodings-2/#notes">More information</a>.</p> <h2 id="upper-level">Upper level<a class="zola-anchor" href="#upper-level" aria-label="Anchor link for: upper-level"><i class="fas fa-link"></i></a> </h2> <p>In order to complete the experiment, we need to check that the original language model done directly at BPE level can be done with this new model-based BPE encoding.</p> <p>It's pretty slow to train that upper level because we need to flow the gradients all the way through the VAE decoder, and the lower layer decoding step, in order to get the <strong>character level loss</strong> (softmax + nll_loss) to properly train something. That's a limit of the current approach.</p> <p>If we randomly split the text into train&amp;validation, we can learn almost perfectly (97% top-1 character level accuracy) the language model on top of that Model based BPE.</p> <img src="/images/models-2-overfit.png"/> <p>However this can be considered <strong>overfitting</strong> because even though a specific input was never seen in the valid set, a very close one <em>was</em>.</p> <p>If instead we try to compare with a fixed split, where the last part of the book is considered the valid set, then we get much lower result.</p> <p>We could achieve 25% exact character matching, and ~77% top-10 character matching on the valid set, which is the end of the book ! The same results happen with BPE, even worse ! we can't get past 13% top-1 and 25% top-10 on the regular BPE. That's understandable because the dataset is very small and the last part of the book is different so it's very hard to infer it from just the beginning and no other text.</p> <p>Another note, is that model based BPE are not tokenizing deterministicly, there is some variance to it, depending on the context of a particular word. This actually seems to be a good property (See <a rel="nofollow external" href="https://arxiv.org/abs/1804.10959">this</a>) and might explain away the better performance of model based BPE over regular BPE. Keep in mind it's 25% of the <strong>characters</strong> that are correct. If we looked at a discrete view of <strong>tokens</strong> we probably would have a much higher prediction rate (it's left for future work for now).</p> <p>Here is a picture from the tensorboard values, P_1 is probability that the character predicted is the correct one, P_10 is that it is in the top-10 values.</p> <img src="/images/models-2-upper.png"/> <p>The overfitting starts happening around the ~1M steps mark.</p> <h3 id="notes">Notes<a class="zola-anchor" href="#notes" aria-label="Anchor link for: notes"><i class="fas fa-link"></i></a> </h3> <ul> <li>In the experiment we learned model by model, freezing the lower model before training something on top. It's because the batching of different layers occur differently. Learning the whole thing end-to-end is probably going to need some thought. The batching is easy for the lower level, every batch needs a tensor of shape CONTEXT_SIZE (=64) of [0-255] ints. For the VAE, we need to have a variable length (depending on the length token) times EMBEDDING_DIM (=128). The upper level needs only tensors of size CONTEXT_SIZE * EMBEDDING_DIM yet if we want to try and end-to-end training, we have <strong>no idea</strong> how many bytes we need to generate 1 correct tensor in the upper layer. We know it's no more than CONTEXT_SIZE² but that would be prohibitive to use that value.</li> <li>The loss NEEDS to always be the byte-level nll loss. At first I thought a simple MSE loss in the embedding space could be enough to learn the proper models. It seems to not be the case. I could only achieve meaningful results by always referring to the original strings and calculating the NLL Loss. When using this loss, the MSE actually <em>increases</em>. This leads me to think that encoding/decoding + softmax are highly anisotropic operators. Looking at the singular values of the embedding matrix, we can see that the highest one is 7.35, the lowest one 0.12, so there are 2 orders of magnitude between the 2. This anisotropy means that the MSE loss which considers all dimensions of the embeddding equal is actually couting way too much some irrelevant dimensions. It would be much faster and simpler if we could train directly on MSE (it would enable us to train without running all the decoding steps to generate the loss). So we need to add some spectral loss on the embedding on the lower language model to test that hypothesis.</li> <li>The tokens have variable lengths. In order to fix this, we have to padd all sequences during learning. Because we padd, we have to mask the padding during training for both VAE and upper LM. Keeping track of this is pretty nifty and it means gradients on rarely used places will rarely get updated. So we will almost surely miss some letters in our tokens. Either at the front or the end of the token depending on how we padd the tokens.</li> </ul> <h2 id="future-work"><strong>Future work</strong><a class="zola-anchor" href="#future-work" aria-label="Anchor link for: future-work"><i class="fas fa-link"></i></a> </h2> <ul> <li>Actually testing discretizing the tokens to compare with the regular BPE. In that direction, also comparing with a randomized tokenizer as used in <a rel="nofollow external" href="https://github.com/google/sentencepiece">SentencePiece</a> to make sure the results are actually comparable and are indeed linked to tokenization variance.</li> <li>The masking problem really seems to be a current limit of the model. Finding a workaround would be really valuable.</li> <li>The fact that the NLL loss is required slows down upper layers. It would be awesome if we could smooth out the encoding/decoding matrix so that L2 directly for VAE and the upper layer works. It probably goes against regular language model embedding so not sure it's doable.</li> <li>Making the epsilon based tokenization directly after the embedding layer. This would help <em>stack</em> those levels hopefully learning higher and higer representations of text leading the sentence embedding and so on.</li> <li>On the same idea, another direction would be to do actual discrete tokenization to allow for the models to stack.</li> </ul> nicolas 2019-05-16T00:00:00+00:00 2019-05-16T00:00:00+00:00 https://nodata.dev/2019/05/16/model-based-bpe-encodings/ <p><a rel="nofollow external" href="https://en.wikipedia.org/wiki/Byte_pair_encoding">Byte-pair encodings</a> (BPE) are now very commonly used in NLP. In <a rel="nofollow external" href="https://openai.com/blog/better-language-models/">GPT-2</a>, Byte-pair encodings are used to preformat the raw texts before feeding the model. But this is a relatively costly step for your preprocessing and has some limitations. For instance, you have to split your data on spaces if you want your byte pair algorithm to compute in reasonable time.</p> <blockquote> <h1 id="tl-dr-in-this-article-we-present-an-idea-to-generate-byte-pair-encodings-not-based-on-frequency-in-the-dataset-but-on-the-quality-of-the-prediction-of-our-model-this-enables-us-to-predict-multi-word-tokens-like-new-york-and-address-languages-that-don-t-use-spaces-to-split-words">TL;DR In this article we present an idea to generate Byte pair encodings, not based on frequency in the dataset, but on the quality of the prediction of our model. This enables us to predict multi word tokens like "New York" and address languages that don't use spaces to split words.<a class="zola-anchor" href="#tl-dr-in-this-article-we-present-an-idea-to-generate-byte-pair-encodings-not-based-on-frequency-in-the-dataset-but-on-the-quality-of-the-prediction-of-our-model-this-enables-us-to-predict-multi-word-tokens-like-new-york-and-address-languages-that-don-t-use-spaces-to-split-words" aria-label="Anchor link for: tl-dr-in-this-article-we-present-an-idea-to-generate-byte-pair-encodings-not-based-on-frequency-in-the-dataset-but-on-the-quality-of-the-prediction-of-our-model-this-enables-us-to-predict-multi-word-tokens-like-new-york-and-address-languages-that-don-t-use-spaces-to-split-words"><i class="fas fa-link"></i></a> </h1> </blockquote> <h2 id="what-are-byte-pair-encodings">What are Byte Pair Encodings ?<a class="zola-anchor" href="#what-are-byte-pair-encodings" aria-label="Anchor link for: what-are-byte-pair-encodings"><i class="fas fa-link"></i></a> </h2> <p>Byte-pair encodings are a way to compress information from pairs of bytes that will form tokens. Let's take an example :</p> <p>"I love carrots and I love apples."</p> <p>This sentence read by a computer is only a sequence of bytes (bytes are simply a number between 0 and 255). That means to a computer our sentence looks like</p> <p>"I love carrots and I love apples." -&gt; [73, 32, 108, 111, 118, 101, 32, 99, 97, 114, 114, 111, 116, 115, 32, 97, 110, 100, 32, 73, 32, 108, 111, 118, 101, 32, 97, 112, 112, 108, 101, 115, 46]</p> <p>From that example, you may remark that some bytes are occurring multiple times together like [108, 111] that occurs twice (it's "lo" from "love"). So let's build a new token for this frequent pair. Numbers from 0 to 255 are already taken so we'll take the next available number which is 256, and we are going to store that information in a table</p> <p>[108, 111] -&gt; 256</p> <p>Now if we use that new token to encode our original bytes, whenever we encounter [108, 111], we'll replace that by 256, so the original byte string becomes :</p> <p>[73, 32, 108, <strong>256</strong>, 101, 32, 99, 97, 114, 114, 111, 116, 115, 32, 97, 110, 100, 32, 73, 32, <strong>256</strong>, 118, 101, 32, 97, 112, 112, 108, 101, 115, 46]</p> <p>We went from 33 numbers to 31 numbers. We can rinse and repeat to compress the number of numbers even further. Originally, BPE was proposed as a compression algorithm. It's not the best compression tool, so we won't look at that side of the algorithm. Now you get what we are looking at when we train a model on BPEs, just a list of numbers.</p> <p>Typically a BPE vocabulary contains ~10k tokens (GPT-2 has 50k), that means it can capture very frequent words like "the" entirely, and parts of words that contain many variations like "ment" (<strong>ment</strong>ally, environ<strong>ment</strong> …). What's great about it it that you can now have words share semantic parts of them for their representation in your model so (environ-ment, environ-ment-al, environ-ment-ally will all share "environ" which will contain most of the semantic meaning, the rest will contain grammar information hopefully).</p> <p>The real advantage of BPE over classical Word Embeddings is that it does not fall into the out-of-vocabulary error (when a word was not seen). At worse you can always fall back to single bytes.</p> <h2 id="what-s-the-problem-with-bpe"><strong>What’s the problem with BPE ?</strong><a class="zola-anchor" href="#what-s-the-problem-with-bpe" aria-label="Anchor link for: what-s-the-problem-with-bpe"><i class="fas fa-link"></i></a> </h2> <p>BPE algorithm is pretty bad in terms of complexity to calculate (roughly O(n²), you can look at a very good implementation <a rel="nofollow external" href="https://github.com/glample/fastBPE">https://github.com/glample/fastBPE</a>). BPE is also pretty bad when you want to encode some new text. A greedy algorithm will be O(n) but not the best encoding possible, the best encoding possible is actually O(n²) in the general case.</p> <p>To be honest, most implementations split on spaces as mentioned earlier which speeds up the algorithm quite a bit. Once we have encoded a full word like “the” there is no way to add tokens to it, so it’s not necessary to look at it anymore for potential byte pairs, so we can assume the encoding&amp;table creation go from O(n²) to something much closer to O(n). In addition, at encoding time, once we know the encoding for “the” we can cache that information leading to further speed ups. But using spaces as a special character has drawbacks, namely:</p> <ul> <li> <p>We can’t address as well languages that don’t use a space to separate words like Chinese (arguably German).</p> </li> <li> <p>We can’t encode frequently occurring multi words like “New York” or “European Union” or “black holes”</p> </li> </ul> <p>The second problem is especially bad when you consider examples where semantic is very different from the composing words like “Chicago Bulls” have nothing to do with bulls.</p> <h2 id="e-bpe-or-model-based-bpe-encoding"><strong>ε-BPE or model based BPE encoding</strong><a class="zola-anchor" href="#e-bpe-or-model-based-bpe-encoding" aria-label="Anchor link for: e-bpe-or-model-based-bpe-encoding"><i class="fas fa-link"></i></a> </h2> <p>The core idea is that instead of using frequency in the dataset to create the byte pairs, we can use the probability transition of the model to create the BPE. Let’s use some kind of transformer, GPT-2 for instance. The core idea of that model, is to predict the next token (in the BPE sense) given a fixed context size. But we can use the output probability of the model in order to create new tokens, not because they are frequent but because they are easy to predict. For instance in a book that contains a character "Sir Francis" that appears rarely, but there is only one character named "Sir …", the algorithm might learn quite easily that "Sir " is followed by "Francis" with great confidence, even if the occurence of the words is pretty low compared to common words like "the", "like" and "I".</p> <p>So the core algorithm, will train a simple transformer on a dataset on regular bytes (at least at the start). Then, as the algorithm learns, some predictions will be above 1-ε. We can keep track of those and keep track of the last token we received, to check if we were correct.</p> <p>Let's keep a hit map to see how successful our algorithm is. For instance, I predicted "Fo" will be followed by "gg" (Phileas Fogg is a character in Around the world in 80 days) with probability &gt; 1-ε. I was correct in 14 cases, and got it wrong in 1 case (let's say it was classical "Fo" "g "). We were correct 14/15 times that's 93% accuracy. If we look at the fluctuation interval associated with that, we get [92.74-93.25%] range. If 92.74 &gt; 1–ε we can conclude that our transition prediction is really very good, it's not a fluke of the model.</p> <p>More generally, if we want 95% confidence when we upgrade this transition, we need to respect the following inequality : k / n - 1/sqrt(n) &gt; 1-ε, where k is the number of successful predictions, n is the total number of predictions and ε the probability margin explained earlier.</p> <p>This model is slightly different from byte pair encoding, but now we don’t suffer from the 2 problems mentioned above, we can get pretty long tokens if the dataset allows for it, and we can use Chinese or German as the space character does not play any special role.</p> <h2 id="results"><strong>Results</strong><a class="zola-anchor" href="#results" aria-label="Anchor link for: results"><i class="fas fa-link"></i></a> </h2> <p>Implementation can be found here. On the first run, we ran on a book <a rel="nofollow external" href="https://en.wikipedia.org/wiki/Around_the_World_in_Eighty_Days">Around the world in 80 days</a> by Jules Verne. It’s a very small dataset but the idea is to check that we can actually overcome BPE’s limitations. Here are a few telling tokens that were created while running on the dataset :</p> <table><thead><tr><th>Promotion #</th><th>Token created</th></tr></thead><tbody> <tr><td>338</td><td>"Mr. Fogg"</td></tr> <tr><td>357</td><td>"Phileas Fogg"</td></tr> <tr><td>360</td><td>"Passepartout"</td></tr> <tr><td>635</td><td>"ir Franc" (Sir Francis)</td></tr> <tr><td>781</td><td>"It was"</td></tr> <tr><td>900</td><td>'" asked' (contains a quote character)</td></tr> </tbody></table> <p>What is interesting, it that:</p> <ul> <li> <p>We managed to create multi word tokens like “Phileas Fogg”</p> </li> <li> <p>Multi word tokens are a minority in terms of tokens created by the algorithm. Out of 421 tokens that contain a space character only 27 are multi word tokens like “New York”. The remaining 394 tokens contain an ending space, meaning our algorithm is learning word boundaries. It is reassuring because traditional BPE are usually hardcoding that information.</p> </li> <li> <p>Multi word tokens are name of characters in the book, which are occurring frequently, they are an entity by themselves (Fogg even has 2 tokens associated to him)</p> </li> <li> <p>2 Multi word tokens are <strong>not</strong> specific to the book, “it was” is a pretty common 2 word token in English in descriptions, “(…) asked” is a very common continuation when we start a quote and end a sentence with a question mark. We can guess that “(…) said” would be a token further down the line, but it’s harder as there are probably a wider variety of verbs that can fit (said, replied, answered and so on…)</p> </li> </ul> <p>Here is a more complete comparison of standard BPE with ε-BPE, with the first 100 tokens generated, as you can see more tokens are dedicated to syntax in eBPE, which Standard BPE ignore gladly by splitting on newlines and spaces.</p> <table><thead><tr><th>Standard BPE</th><th>eBPE</th></tr></thead><tbody> <tr><td>'th'</td><td>'\r\n'</td></tr> <tr><td>'the '</td><td>', '</td></tr> <tr><td>'an'</td><td>'d '</td></tr> <tr><td>'in'</td><td>'Th'</td></tr> <tr><td>'ou'</td><td>'ve'</td></tr> <tr><td>'er'</td><td>'y '</td></tr> <tr><td>'ed '</td><td>'; '</td></tr> <tr><td>'ar'</td><td>'f '</td></tr> <tr><td>'hi'</td><td>',\r\n'</td></tr> <tr><td>'on'</td><td>'\r\n\r\n'</td></tr> <tr><td>'re'</td><td>'th'</td></tr> <tr><td>'en'</td><td>'qu'</td></tr> <tr><td>'and '</td><td>'the'</td></tr> <tr><td>'of '</td><td>' '</td></tr> <tr><td>'st'</td><td>'the '</td></tr> <tr><td>'to '</td><td>'The'</td></tr> <tr><td>'as '</td><td>'\r\n'</td></tr> <tr><td>'se'</td><td>', '</td></tr> <tr><td>'ha'</td><td>'y '</td></tr> <tr><td>'or'</td><td>'d '</td></tr> <tr><td>'.\r '</td><td>'Th'</td></tr> <tr><td>'it'</td><td>'ve'</td></tr> <tr><td>'he '</td><td>'; '</td></tr> <tr><td>'le'</td><td>'f '</td></tr> <tr><td>'ing '</td><td>',\r\n'</td></tr> <tr><td>',\r '</td><td>' '</td></tr> <tr><td>'as'</td><td>'\r\n'</td></tr> <tr><td>'in '</td><td>', '</td></tr> <tr><td>'at'</td><td>'d '</td></tr> <tr><td>'at '</td><td>'y '</td></tr> <tr><td>'ro'</td><td>'Th'</td></tr> <tr><td>'er '</td><td>'ve'</td></tr> <tr><td>'al'</td><td>'f '</td></tr> <tr><td>'es'</td><td>'; '</td></tr> <tr><td>'on '</td><td>' '</td></tr> <tr><td>'was '</td><td>',\r\n'</td></tr> <tr><td>'no'</td><td>'th'</td></tr> <tr><td>'his '</td><td>'\r\n'</td></tr> <tr><td>'ed'</td><td>', '</td></tr> <tr><td>'ac'</td><td>'d '</td></tr> <tr><td>'"\r '</td><td>'y '</td></tr> <tr><td>'ri'</td><td>'Th'</td></tr> <tr><td>'be'</td><td>'ve'</td></tr> <tr><td>'ly '</td><td>'f '</td></tr> <tr><td>'om'</td><td>'; '</td></tr> <tr><td>'li'</td><td>' '</td></tr> <tr><td>'en '</td><td>',\r\n'</td></tr> <tr><td>'ti'</td><td>'th'</td></tr> <tr><td>'og'</td><td>'\r\n\r\n'</td></tr> <tr><td>'ra'</td><td>'the'</td></tr> <tr><td>'di'</td><td>'the '</td></tr> <tr><td>'art'</td><td>'The'</td></tr> <tr><td>'Fog'</td><td>'qu'</td></tr> <tr><td>'the'</td><td>'s '</td></tr> <tr><td>'ma'</td><td>'The '</td></tr> <tr><td>'ve '</td><td>'g '</td></tr> <tr><td>'is '</td><td>',"'</td></tr> <tr><td>'or '</td><td>'no'</td></tr> <tr><td>'ld '</td><td>'t '</td></tr> <tr><td>'whi'</td><td>'th '</td></tr> <tr><td>'il'</td><td>'o '</td></tr> <tr><td>'ur'</td><td>'?"'</td></tr> <tr><td>'s, '</td><td>'\r\n\r\n"'</td></tr> <tr><td>'de'</td><td>'," '</td></tr> <tr><td>'wh'</td><td>'Mr'</td></tr> <tr><td>'lo'</td><td>'e '</td></tr> <tr><td>'ch '</td><td>'yo'</td></tr> <tr><td>'ere '</td><td>'Yo'</td></tr> <tr><td>'ith '</td><td>'ou'</td></tr> <tr><td>'The '</td><td>'. '</td></tr> <tr><td>'am'</td><td>'nd '</td></tr> <tr><td>'ent'</td><td>'h '</td></tr> <tr><td>'un'</td><td>'n '</td></tr> <tr><td>'gh'</td><td>';\r\n'</td></tr> <tr><td>'with '</td><td>'og'</td></tr> <tr><td>'an '</td><td>'you'</td></tr> <tr><td>'oun'</td><td>'r '</td></tr> <tr><td>'part'</td><td>'of '</td></tr> <tr><td>'ver'</td><td>'to '</td></tr> <tr><td>'si'</td><td>'s F'</td></tr> <tr><td>'had '</td><td>'Pa'</td></tr> <tr><td>'not '</td><td>'as '</td></tr> <tr><td>'ould '</td><td>''s '</td></tr> <tr><td>'ing'</td><td>'. F'</td></tr> <tr><td>'out '</td><td>'is '</td></tr> <tr><td>'el'</td><td>'ld '</td></tr> <tr><td>'sa'</td><td>'ng '</td></tr> <tr><td>'ce'</td><td>'at '</td></tr> <tr><td>'that '</td><td>'re'</td></tr> <tr><td>'asse'</td><td>'ve '</td></tr> <tr><td>'fi'</td><td>'gh'</td></tr> <tr><td>'ol'</td><td>'ut '</td></tr> <tr><td>'sh'</td><td>'ll'</td></tr> <tr><td>'r. '</td><td>'Pas'</td></tr> <tr><td>'."\r '</td><td>'re '</td></tr> <tr><td>'Passe'</td><td>'ed '</td></tr> <tr><td>'Passepart'</td><td>'. Fog'</td></tr> <tr><td>'ut '</td><td>'ch '</td></tr> <tr><td>'which '</td><td>'and '</td></tr> <tr><td>'ay'</td><td>'ea'</td></tr> </tbody></table> <p>I would love to check the tokenization of German or Chinese but I’m not a speaker of either language so it’s hard for me to analyze the results anyway. What’s for sure is that the technique is applicable.</p> <p>I also tried the technique on different types of files like wav files or mp3 files, even jpeg images. Analysis is harder to do. Still some interesting notes, it took longer for the model to emit new tokens on the mp3 files than on the wav files. The mp3 file is encoded, therefore should have a lower entropy (meaning it’s harder to predict the next token) than the wav files so the model takes longer to actually get good at predicting. It’s probable (I haven’t checked) that we have to overfit the mp3 file and jpeg files before we can predict any meaningful content (except maybe the header part)</p> <h2 id="future-work"><strong>Future Work</strong><a class="zola-anchor" href="#future-work" aria-label="Anchor link for: future-work"><i class="fas fa-link"></i></a> </h2> <p>Many interesting ideas are still left to explore to continue exploring the idea of models creating their own tokenization. For now a limiting factor is the actual BPE encoding process that takes longer and longer as the model creates new tokens. That's because the encoding process is done in Python, so it's quite slow and can’t be precalculated as you would do with fixed BPE encodings. To give a sense of the slowdown, the training loop starts at ~11it/s on a GTX970 and finished at roughly 10s/it. That’s a 100x slowdown over the course of the training, with only 1k tokens in the end, far from the 50k used by GPT-2 for instance.</p> <p>It’s going to be an actual requirement to train on larger and more representative datasets. Training on bigger datasets would help us understand how important are those multi word tokens and maybe what are those multi words. The token "(…) <strong>asked</strong>" was pretty surprising to me, I'm eager to see what else can be discovered.</p> <p>The actual epsilon used was 40% which actually quite a big (value was chosen with trial and error, to get a small but not null rejection rate of new tokens, to add tokens as fast as possible but not making too many mistakes). That value probably has a sweet spot depending on the number of current tokens, after speeding up the process it would be interesting to look at the best value for epsilon as a function of the number of tokens.</p> nicolas 2018-04-13T00:00:00+00:00 2018-04-13T00:00:00+00:00 https://nodata.dev/2018/04/13/a-third-of-your-math-homework-is-useless/ <h1 id="a-third-of-your-math-work-is-useless">A third of your math work is useless<a class="zola-anchor" href="#a-third-of-your-math-work-is-useless" aria-label="Anchor link for: a-third-of-your-math-work-is-useless"><i class="fas fa-link"></i></a> </h1> <p>At Kwyk, we use AI to predict student’s performance, especially in maths. Our most recent results show that on <strong>one-third of the exercises</strong> given by the teacher <strong>we know beforehand that the student will succeed</strong>, raising the question of its usefulness. Despite the clickbait title <em>(sorry for that)</em>, the answer might not be as simple as you would expect. We are going to attempt to explain that <strong>the biggest threat in AI is not AI in itself, but our human over-interpretation of its results.</strong></p> <h1 id="introduction">Introduction<a class="zola-anchor" href="#introduction" aria-label="Anchor link for: introduction"><i class="fas fa-link"></i></a> </h1> <p>In this article, we will analyze prediction data done on 1 million maths exercises done between March 7th, 2018 and March, 23rd 2018 on <a rel="nofollow external" href="https://www.kwyk.fr">Kwyk</a>. <strong>The predictions were done in real time.</strong> The training was done on a 5 Million exercises dataset from September and October 2016 results.</p> <p>Kwyk is an online learning platform where students do their homework online. The students are age 11 to 18 and are assigned homework by teachers on the platform. Each individual has a different homework generated automatically, but all questions are equivalent in terms of difficulty : 1/3 + 1/2 is roughly the same as 2/3 + 1/4 for instance. Because homework is generated, students can retry anytime on a new version of the homework to get the best grade. They can also work freely on any given chapter at any times. To be fair, 90% of the activity is work assigned by teachers.</p> <p>We also grade automatically every exercise, using <a rel="nofollow external" href="https://en.wikipedia.org/wiki/Computer_algebra_system">formal analysis</a>. This is how we evaluate if a student succeeded an exercise or not. What we evaluate is the <strong>probability of success of a student on a specific exercise.</strong></p> <h1 id="how-we-got-our-results">How we got our results<a class="zola-anchor" href="#how-we-got-our-results" aria-label="Anchor link for: how-we-got-our-results"><i class="fas fa-link"></i></a> </h1> <p>The results were obtained using an AdaBoost regression, on manually crafted features, which included various averages, including the student’s average on this particular chapter for instance. We also used automatically crafted features, which are sort of the map of maths, which we explain in detail <a rel="nofollow external" href="https://medium.com/data-science/a-non-nlp-application-of-word2vec-c637e35d3668">here</a>. In total there are 10 manual features and 50 automatic features.</p> <h1 id="results-summary">Results summary<a class="zola-anchor" href="#results-summary" aria-label="Anchor link for: results-summary"><i class="fas fa-link"></i></a> </h1> <p>Now the results. First, we get an AUC score of 0.89. Unfortunately, we don’t have an expert reference (different teachers estimating success likelihood of students). Not having comparison points makes it really hard to evaluate if this score is interesting or not. We can only compare to previous versions of our algorithms, which started with AUC 0.72 so we definitely improved our assessment of students, you can read our other articles for details on our algorithms and how we assess them.</p> <p><img src="/assets/images/a-third-1.png" alt="" /> Area Under Curve of our predictions (Prediction data was evaluated in real time)</p> <p>Out of the 1 Million exercises we assessed, we estimated on 300k exercises that the student would succeed over 95% of the times. They succeeded on average over 96% of the time. In other words, our algorithm was able to predict that you knew how to do this exercise a third of the time. <strong>If this is so, was it really necessary for you to do it ?</strong> The temptation to answer no is exactly the problem that AI poses to us, humans.</p> <p>On the other end of the spectrum, we estimated on 100k exercises that the student would succeed less than 5% of the times. In reality, they succeeded 10% of the times. So our algorithm is less accurate on low probabilities. This is expected as we don't control what happens between exercises, so if a student asks for help or reviews his notes before answering, we have no way of capturing that in our data.</p> <p>But in the end, it means that we roughly know a student’s result in advance on 40% of the questions asked by his teacher.</p> <p>Overall you can see how precise is our algorithm on the following graph</p> <p><img src="/assets/images/a-third-2.png" alt="" /></p> <p>Each dot is a group of predictions. The closer the dots are to the blue line, the better the prediction was for that group. The two big dots at each end are the one mentioned in this article.</p> <blockquote> <p>With AI we know a student's result in advance on 40% of the questions asked by his teacher.</p> </blockquote> <h1 id="discussion">Discussion<a class="zola-anchor" href="#discussion" aria-label="Anchor link for: discussion"><i class="fas fa-link"></i></a> </h1> <p>So 40% of the time we know in advance with a small margin of error if a student is going to succeed at a specific exercise. Why is it that we should NOT conclude that it was a waste of time. There are two types of reasons for this.</p> <p><strong>The first type of reasons is over-interpretation of the results.</strong> The algorithm used tried to assess the likelihood of success of a given student. It was NOT to assess if it was necessary for this student to answer this question (which by the way is a much more difficult question to ask an algorithm). AI works by using correlations. So imagine a new student being transferred to an excellent class that never makes any mistake. An algorithm will most likely categorize this student as being very likely to succeed. This new student will be categorized as "no need to work" even if he did not do any exercise yet. This is obviously wrong.</p> <p>The error is not in the algorithm, as most students that get transferred into excellent classes will be excellent students, the error is in us humans, that over-interpret what the algorithm says. The likelihood of success is not the proven mastery of a given subject, yet we humans will tend to think that both are the same because we estimate the likelihood of success with our estimate of a student's mastery. AI does not work that way. Not understanding this is really the core of the problem with AI. Because we cannot ask our algorithms questions like humans, we need to find proxy questions that we hope will be close to our real questions. In our case the real question we would like to ask is "What does this student know", and the proxy we found is "How likely is he to succeed at this task".</p> <blockquote> <p>AI forces us to change our real world vague questions into extremely specific proxy questions. The human confusion between the two questions is the biggest threat to integrating AI into our products.</p> </blockquote> <p>The same problem is found over and over. Facebook question should be "How to maximize our users' happiness", and the proxy they use is "How to maximize view time on our site". Google's question is "What is the answer to this user's query" and its proxy is "What is the link with the lowest bounce rate on this user's query ?". Understanding that there is a fundamental gap between the question we would like to ask our algorithm and the actual proxy we use is extremely important if we want to avoid errors in the future.</p> <p><strong>The second type is related to neuroscience and is specific to this problem</strong>. Suppose now that our proxy is perfectly valid to estimate a student's mastery. Our prediction still does not take into account our human brains. We forget all the time. In order to memorize something, we need to recall it many times. <a rel="nofollow external" href="https://en.wikipedia.org/wiki/Spaced_repetition">Spaced repetition</a> is an example of how to take this into account. So even if you already know something, if you want to fix it in your memory you need to do it over and over at regular intervals. Once again, even if our algorithm knows that you know something in advance, it might be in your best interest to do the exercise anyway to fix the knowledge in your brain.</p> <p>In addition to <em>memory</em>, the <em>student's emotional state</em> is another variable that we might want to take into account when we look at the bigger picture of education.</p> <h1 id="how-to-prevent-this-over-interpretation">How to prevent this over-interpretation<a class="zola-anchor" href="#how-to-prevent-this-over-interpretation" aria-label="Anchor link for: how-to-prevent-this-over-interpretation"><i class="fas fa-link"></i></a> </h1> <p>Exactly like the <a rel="nofollow external" href="https://en.wikipedia.org/wiki/Simpson%27s_paradox">Simpson's paradox,</a> there is no silver bullet here. The main way to prevent over-interpreting is to be extremely conscious of the problem. We, as practitioners, need to be aware of the questions we asked our algorithm and to be extremely careful not to over-interpret the answers they give us. We also need to alert everyone else to the types of conclusions that should not be drawn from our results.</p> <p>We can also multiply the different questions to ask our algorithms. If many specific algorithms seem to converge on their answers, maybe it is a hint that it could be a good answer to our human questions.</p> <p>If only we could add dropout to our interpretations…</p> <h1 id="further-work">Further work<a class="zola-anchor" href="#further-work" aria-label="Anchor link for: further-work"><i class="fas fa-link"></i></a> </h1> <p>At Kwyk we have many goals in order to improve overall student's success. In addition to ever improving our student's success estimate, we need to try and address the problems evoked in this article.</p> <p>For instance, we are looking to take into account the sequencing of personalized sequences (the second type of problems). For this we still cannot ask the human questions, we need to find a proxy question. The proxy question will be "What is the best strategy of homework to give a specific student to maximize his future likelihood of success on all exercises ?". For this, we will use RL approaches. Of course, we will necessarily encounter problems along the road but we hope we can come closer and closer to answering our real question which is "How do we maximize each student's success ?".</p> nicolas 2017-07-31T00:00:00+00:00 2017-07-31T00:00:00+00:00 https://nodata.dev/2017/07/31/a-non-nlp-application-of-word2vec/ <p>When using <strong>Machine Learning</strong> to solve a problem, having the right <strong>data</strong> is crucial. Unfortunately, raw data is often “unclean” and <strong>unstructured</strong>. <a rel="nofollow external" href="https://en.wikipedia.org/wiki/Natural_language_processing">Natural Language Processing</a> (<strong>NLP</strong>) practitioners are familiar with this issue as all of their data is <strong>textual</strong>. And because most of Machine Learning algorithms can’t accept raw <em>strings</em> as inputs, <strong>word embedding</strong> methods are used to transform the data before feeding it to a learning algorithm. But this is not the only scenario where textual data arises, it can also take the form of categorical features in standard non-NLP tasks. In fact, many of us struggle with the processing of these kinds of features, so are word embedding of any use in this case ?</p> <p>This article aims to show how we were able to use <strong>Word2Vec</strong> (<a rel="nofollow external" href="https://arxiv.org/pdf/1301.3781.pdf">2013, Mikolov et al.</a>), a word embedding technique, to convert a <strong>categorical feature</strong> with a <strong>high number of modalities</strong> into a smaller set of easier-to-use <strong>numerical features</strong>. These features were not only easier to use but also successfully <strong>learned relationships</strong> between the several modalities similar to how classic word embeddings do with language.</p> <h1 id="word2vec">Word2Vec<a class="zola-anchor" href="#word2vec" aria-label="Anchor link for: word2vec"><i class="fas fa-link"></i></a> </h1> <blockquote> <p>You shall know a word by the company it keeps (<a rel="nofollow external" href="https://en.wikipedia.org/wiki/John_Rupert_Firth">Firth, J. R.</a> 1957:11)</p> </blockquote> <p>The above is exactly what Word2Vec seeks to do : it tries to determine the meaning of a word by analyzing its neighboring words (also called <strong>context</strong>). The algorithm exists in two flavors <strong>CBOW</strong> and <strong>Skip-Gram</strong>. Given a set of sentences (also called <strong>corpus</strong>) the model loops on the words of each sentence and either tries to use the current word of to predict its neighbors (its context), in which case the method is called “Skip-Gram”, or it uses each of these contexts to predict the current word, in which case the method is called “Continuous Bag Of Words” (CBOW). The limit on the number of words in each context is determined by a parameter called “<strong>window size</strong>”.</p> <p><img src="/assets/images/non-nlp1.png" alt="" /></p> <p>Both Word2Vec architectures. The current word is w(t) and w(t-2)..w(t+2) are context words. (Mikolov et al. 2013)</p> <p>So if we choose for example the Skip-Gram method, Word2Vec then consists of using a <strong>shallow neural network</strong>, i.e. a neural network of only one hidden layer, to learn the word embedding. The network first initializes randomly its weights then iteratively adapt these during training to minimize the error it makes when using words to predict their contexts. After a hopefully successful training, the word embedding for each word is obtained by multiplying the network’s <strong>weight matrix</strong> by the word’s <a rel="nofollow external" href="https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science">one-hot vector</a>.</p> <blockquote> <p>Note : Besides allowing for a numerical representation of textual data, the resulting embedding also learn interesting <a rel="nofollow external" href="http://www.aclweb.org/anthology/N13-1090"><strong>relationships between words</strong></a> and can be used to answer questions such as : <em>king is to queen as father is to …?</em></p> </blockquote> <p>For more details on Word2Vec you can have a look at this <a rel="nofollow external" href="https://www.youtube.com/watch?v=ERibwqs9p38">Stanford lecture</a> or this <a rel="nofollow external" href="https://www.tensorflow.org/tutorials/word2vec">tutorial by Tensorflow</a>.</p> <h1 id="application">Application<a class="zola-anchor" href="#application" aria-label="Anchor link for: application"><i class="fas fa-link"></i></a> </h1> <p>At <a rel="nofollow external" href="https://www.kwyk.fr/"><strong>Kwyk</strong></a> we provide math exercises online. Teachers assign <strong>homework</strong> to their students and each time an exercise is done some <strong>data</strong> is stored. Then, we use the collected data to <strong>evaluate the students’ level</strong> and give them <strong>tailored</strong> review exercises to help them progress. For each exercise that is answered we store a list of identifiers that help us tell : <strong>what is the answered exercise ?</strong>, <strong>who is the student ? , what is the chapter ?</strong>… In addition to that, we store a <strong>score</strong> value that is either (0) or (1) depending on the student success. To evaluate the students’ levels we then simply have to predict this score value and get success probabilities from our classifier.</p> <p>As you can see, a lot of our features are <strong>categorical</strong>. Usually, when the number of <strong>modalities</strong> is small enough, one can simply transform a categorical feature with (n) modalities into (n-1) dummy variables then use that for training. But when the number of modalities is in the many thousands — as it is the case for some of our features — relying on dummy variables becomes inefficient and impracticable.</p> <p>In order to address this issue our idea is to use Word2Vec to <strong>transform categorical features</strong> into a relatively small number of usable <strong>continuous features</strong> using a little trick. To illustrate, let’s consider “exercise_id”, a categorical feature telling us which exercise was answered. In order to be able to use Word2Vec we have to provide a <strong>corpus</strong>, a set of sentences to feed to the algorithm. But the raw feature — a list of ids — isn’t a corpus <em>per se :</em> the order is completely <strong>random</strong> and closer ids don’t carry any <strong>information</strong> about their neighbors. <strong>Our trick</strong> consists of considering each homework given by a teacher as a “sentence”, a <strong>coherent</strong> list exercise ids. As a result, ids are <strong>naturally</strong> gathered by levels, chapters… and Word2Vec can start learning exercise embedding directly on that.</p> <p>Indeed, thanks to these artificial sentences we were able to use Word2Vec and get beautiful results :</p> <p><img src="/assets/images/non-nlp2.png" alt="" /></p> <p>Exercise embedding (3 main components of PCA) colored by level. 6e, 5e, 4e, 3e, 2e, 1e and tm are the french equivalents of the 6th, 7th, 8th, 9th, 10th, 11th and 12th grades in the US.</p> <p>As we can see, the resulting embedding has a <strong>structure</strong>. In fact, the 3d-projected cloud of exercises is spiral-shaped with <strong>exercises of higher levels following directly those of previous levels</strong>. This means that the embedding successfully learned to <strong>distinguish</strong> exercises of different school levels and regrouped similar exercises together. But that’s not all, using a <a rel="nofollow external" href="https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction#Locally-linear_embedding"><strong>non-linear dimensionality reduction</strong></a> technique we were able to reduce the whole embedding into a single real valued variable with the same characteristics. In other terms, we obtained an <strong>exercise complexity</strong> feature that is minimum for 6th grade exercises and <strong>grows</strong> as the exercises get more and <strong>more complex</strong> until it is maximum for 12th grade exercises.</p> <p>Moreover, the embedding also learned <strong>relationships</strong> between exercises just like Mikolov’s did with English words :</p> <p><img src="/assets/images/non-nlp3.png" alt="" /></p> <p>The diagram above shows some examples of the relationships our embedding was able to learn. So if we were to ask the question <em>“an exercise of</em> <strong><em>number addition</em></strong> <em>is to an exercise of</em> <strong><em>number subtraction</em></strong> <em>as an exercise of</em> <strong><em>time addition</em></strong> <em>is to … ?”</em> the embedding gives us the answer “<em>an exercise of</em> <strong><em>time subtraction</em></strong>”. Concretely, this means that if we take the difference embedding[Substract(Numbers)] - embedding[Add(Numbers)] and add it to the embedding of an exercise where students are asked to add time values (hours, minutes …) then the closest embedding is one of an exercise that consists of subtracting time values.</p> <h1 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion"><i class="fas fa-link"></i></a> </h1> <p>All in all, word embedding techniques are useful to transform textual data into real valued vectors which can then be plugged easily into a machine learning algorithm. Despite being principally used for NLP applications such as <em>machine translation</em>, we showed that these techniques also have their place for categorical feature processing by giving the example of a particular feature we use at Kwyk. But in order to be able to apply a technique such as Word2Vec, one has to build a corpus — i.e. a set of sentences where labels are arranged so that a context is implicitly created. In our example, we used homework given on the website to create “sentences” of exercises and learn an exercise embedding. As a result we were able to get new numeric features that successfully learned the relationships between exercises and are then more useful than the bunch of labels they originated from.</p> <p><em>Credits go to Christophe Gabard, one of our developers at Kwyk, for having the idea of applying Word2Vec to process categorical features.</em></p>