scala optimization techniques

Share on … Optimization Techniques; Learning Objectives: Understand various optimization techniques like Batch Gradient Descent, Stochastic Gradient Descent, ADAM, RMSProp. Subskills. Typically, when you are micro-optimizing a library like Fansi, you spend time with a profiler and see what takes up the most time. Optimization results. From this profile, we can see where the time is going when we run our code: And from there, you figure out ways to make the code run faster. Nevertheless, one thing is clear: the Parsing performance has dropped by half, again! For distributed environment- and cluster-based ... Take O’Reilly online learning with you and learn anywhere, anytime on your phone and tablet. In this course, we cut the weeds at the root. In this section, we will discuss how we can further optimize our Spark applications by applying data serialization by tuning the main memory with better memory management. Debug Apache Spark jobs running on Azure HDInsight DESCRIPTION - With attributes describing various aspect of residential homes, you are required to build a regression model to predict the property prices using optimization techniques like gradient descentT. Tags: optimization, spark. Optimization techniques There are several aspects of tuning Spark applications toward better optimization techniques. deconstructed the complexity of Spark in bite-sized chunks that you can practice in isolation selected the essential concepts and exercises with the appropriate complexity sequenced the topics in increasing order of difficulty so that they "click" along the way applied everything in live code 13 hours ago How to write Spark DataFrame to Avro Data File? If you find yourself using Arrays for performance reasons, .copyOfRange is definitely something that's worth thinking of! That's something turning from "noticeable lag" to "annoying delay". By pre-filling the lookupAttrTable array, we can make the lookup really fast, without wasting any space storing huge, empty Arrays. The L-BFGS method approximates the objective function locally as a quadratic without evaluating the second partial derivatives of the … Typically, we would reach for a Map[String, T] first. If it's some internal webpage that someone looks at once every-other week, then maybe not. Simulation packages : tableau , event , process , dynamics , dynamics_pde , activity , state . The colors array stores Str.States, which is really just a type-alias for Int. Also, offers to build an extensible query optimizer. Otherwise, the ignorance of them can lead to inefficient run times and system downtimes. Maths for Optimization; Optimization Strategies; Delivery Type: Theory. The book is only 274 pages so it can feel pretty small. Strategic Scala Style: Designing Datatypes, Next (8%) is initializing iterators, again inside, Iterating over ever key/value in the map and checking if it matches starting at, Using a regex to try and pull out the Ansi color code, before putting it into the map. gcd(14, 21)is evaluated as follows: Now, consider factorial: factorial(4)is evaluated as follows: What are the differences between the two sequences? Let’s take a look at these two definitions of the same computation: Lineage (definition1): Lineage (definition2): The second definition is much faster than the first because i… Intermediate. "Fast enough" could mean "fast enough, if you're careful", but with extra performance it could be "fast enough, no need to care at all" and save you some headache. So we can be confident that despite being implemented totally differently, the externally-visible behavior is exactly the same. Scala: Mathematical Optimization Time for a math lesson! Furthermore, catalyst optimizer in Spark offers both rule-based and cost-based optimization as well. This post will demonstrate the potential benefit of micro-optimizations, and how it can be a valuable technique to have in your toolbox of programming techniques. This is slow to run, and error prone: if you forget to remove them, you end up with subtle bugs where you're treating a string as if it is 27 characters long on-screen but it's actually only 22 characters long since 5 characters are an Ansi color-code that takes up no space. Although it only took about 50 characters to implement, it isn't something that a typical Scala programmer would reach for out of the box. times faster, and have made it take ~6.3x less memory to store its data-structures. Skills ML. In this case, I just need to iterate over the Categorys that are available, and there's no faster data-structure to iterate over than a flat Array. See the linked talk in the comments for details on that. Although allocating this array costs something, it's the Attr.categories vector only has 5 items in it, so allocating a 5-element array should be cheap. 2.1. Let's see how to calculate minimum or maximum values of equations in Scala with some help from the Optimus library. As a real-world use case to demonstrate these techniques, I am going to use the Fansi library. For example, storing our Str.State in a bit-packed Int rather than a Map[Category, Attr] makes it blazing fast, but it also means that: Library-users cannot define their own Attrs: they have to be known in advance, and fit nicely into the bit-ranges assigned to them. We dive deep into Spark and understand what tools you have at your disposal - and you might just be surprised at how much leverage you have. In this case we did the second option, and here's how the numbers look: Again, there is a great deal of noise in these results. Here, an in-memory object is converted into another format that can be stored in … The huge slowdown to Overlay is not unexpected: after all, we do the most of our heavy lifting regarding Str.State inside .overlay, where we need to apply the modifications to the state of every character our Attrs are being overlayed on. Hence, by looking up the Attr via it's applyMask >> offset, we are able to keep the lookup to a relatively integer, in the hundreds. . The applyMask and resetMask for combinations of Attrs can be computed from those of each individual Attrs object. If our code is taking 0.1ms out of a batch process that takes 10 minutes to run, it's certainly not worth bothering to optimize. Each node has a node type and zero or more children. Optimization techniques There are several aspects of tuning Spark applications toward better optimization techniques. To pass we have to aggressively throw out-of-bounds exceptions ourselves: It turns out that this works, and all the test cases pass, but at a cost of some performance: There's some noise in this measure, as you'd expect: Rendering and Concat has seemed to have gotten faster. Stacks and Stack Frames. The Fansi library has already been optimized, and thus I have already gone through this process, identified the various bottlenecks, and optimized them one by one. The next micro-optimization we can try removing is the local categoryArray variable: This was introduced to make the while-loop going over the Attr.categories vector faster inside the render method. Bit-packing is a technique that is often ignored in "high level" languages like Scala, despite having a rich history of usage in C++, C, or Assembly programs. Get Scala and Spark for Big Data Analytics now with O’Reilly online learning. This post will use the Fansi library as a case-study for what benefits you get from micro-optimizing Scala: swapping out elegant collection transformations for raw while-loops over mutable Arrays, elegant case classs for bit-packed integers. In the case of Fansi, it probably is: the Ansi color codes haven't changed much for decades and are unlikely to start changing quickly now. Now that we've gone through roughly how Fansi works, we will start with the step-by-step de-optimization of the library, removing the existing micro-optimizations one by one and seeing how the performance is affected by each one. Hence, the resetMask of Attrs tells you which bit-ranges need to be cleared in order for the Attrs to be applied, and the applyMask tells you what those bit-ranges will get set to. At all points throughout this post, as the various optimizations are removed one by one, the full test suite is passing. Data Serialization The result optimization is typically between 150 KB and a few hundreds of KB. To a user, that's something turning from "instant" to "noticeable lag". There are several aspects of tuning Spark applications toward better optimization techniques. From Scala source files to optimized JavaScript code, there are a few steps which are described in this document. Things like removing intermediate objects to minimize memory allocations, or using bit-sets rather than HashSets to speed up lookups, are examples of micro-optimizations. This is a question everyone asks: is it worth putting in the effort to micro-optimize something and mess up the code a little, in exchange for the performance gain? TITLE - Classify good and bad customer for bank to decide on granting loans That is definitely a loss of flexibility and extensibility. Let's see how to calculate minimum or maximum values of equations in Scala with some help from the Optimus library. And ~8.5 times as much memory as the colored java.lang.Strings. What's the take-away? One of the most important aspects is garbage collection, and it's tuning if you have written your Spark application using Java or Scala. for setting the background color via Back.LightGreen, Nevertheless, much of the size of that integer is due to the offset of the category to stop it from overlapping with others; in this case, for example, the applyMask of Back.LightGreen can only really start after the twelfth bit (the area which the resetMask covers). Micro-optimization has a bad reputation, and is especially uncommon in the Scala programming language where the community is more interested in other things such as proofs, fancy usage of static types, or distributed systems. If you’re interested in other Scala-related articles based on the experiences of Threat Stack developers, have a look at the following: Useful Scala Compiler Options, Part 2: Advanced Language Features; My Journey in Scala, Part 1: Awakenings; My Journey in Scala, Part 2: Tips for Using IntelliJ IDEA It stores its characters and their colors in two parallel Arrays. Nevertheless, sometimes you find your code is spending a significant amount of time in one section, and you want it to spend less. Welcome to the fourteenth lesson ‘Spark RDD Optimization Techniques’ of Big Data Hadoop Tutorial which is a part of ‘Big Data Hadoop and Spark Developer Certification course’ offered by Simplilearn. This is a tiny library that I wrote to make it easier to deal with color-coded Ansi strings: This library exists because dealing with raw java.lang.Strings with Ansi escape codes inside is troublesome, slow and error-prone. Furthermore, storing all the data relevant to the current state requires only 32 bits, far less than would be required to store a hash-table or tree or whatever data-structures a Set requires. It shares all the properties of java.lang.String, for better or worse. "Micro-optimization" is normally used to describe low-level optimizations that do not change the overall structure of the program; this is as opposed to "high level" optimizations (e.g. These changes can often be made entirely local to a small piece of code, leaving the rest of your codebase untouched. I always recommend the practical approach to learning and Scala in Action is the … Nevertheless, people often write for-loops naturally and only optimize it later. You will learn 20+ techniques and optimization strategies. I posted it here because I am looking for practical and scala-specific advice and not theorical and generic optimization advice. What are your favorite micro-optimization tricks you've used in Scala or other languages? Stacks. If it's taking 300ms out of the 600ms that our webserver takes to generate a response, is it worth it then? Spark optimization techniques are used to modify the settings and properties of Spark to ensure that the resources are utilized properly and the jobs are executed quickly. In the case of Fansi, after optimization the above profile turns into: At which point, all our time is being spent inside this render method, and not in any other helpers or auxiliary code. The .render method serializes this into a single java.lang.String with Ansi escape-codes embedded. The first step of making this "idiomatic" or "typical" Scala is to replace all our usage of System.arraycopy and java.util.Arrays. choosing efficient algorithms, caching things, or parallelizing things) that often … The benefit of this data-structure is that doing operations on the Str is really fast and easy: Without having to worry about removing Ansi codes first, or having our colors get mixed up as we slice and concatenate them. Attribute(name: String):an attribute from a… Let’s compare the evaluation steps of the application of two recursivemethods. Do you need to design your application to avoid doing redundant work? If it's taking 9 minutes out of the 10 minutes a process takes to run, it's more likely to be worth it. 13 hours ago How to read a dataframe based on an avro schema? The goal of Fansi is to make such mistakes impossible, and to have such simple operations behave as you'd expect with regard to colors: The Fansi documentation has a lot more to say about why Fansi exists, but this should have given you a flavor of the problem it's trying to solve. These are non trivial performance gains to be had; but are they worth the cost? Note we did not change the while loop in the Str.apply method we use to parse the fansi.Strs out of java.lang.Strings: This while-loop skips forward by varying numbers of characters each iteration, and cannot be changed into a trivial for-loop like the others. In general, even when performance is "fast enough", you an often benefit from parts of your code having higher performance: if you don't need the speed, you can often trade off speed against convenience. So far we've been removing one optimization at a time and seeing what happens. It is based on functional programming construct in Scala. To understand functional loops in Scala, it’s important to understand what stacks are. It is the process of converting the in-memory object to another format … Furthermore, all these operations are implemented as fast System.arraycopys and Arrays.copyOfRanges: Which perform much faster than copying the data yourself using a for-loop or Scala collections operations like .drop and .take. One bit of unusual code is the val lookupAttrTable: Array[Attr] that's part of the Category class, The purpose of this method is to make it quick to look up an Attr based on its .applyMask. You are looking at the only course on the web which leverages Spark features and capabilities to the max. red being \u001b[31m, underlined \u001b[4m, and remove all of them before being counting the length. We’ll also look into common pitfalls to avoid as well as optimization techniques to help make our code concise as well as avoid running into errors. But it moves fast and covers a lot of ground with Scala performance. Optimization Techniques in Spark (i)Data Serialization - Java Serialization, Kyro serialization (ii)Memory Tuning - Data Structure tuning, Garbage collection tuning (iii)Memory Management - Cache() and Persist() With the techniques you learn here you will save time, money, energy and massive headaches. We will look at how we can also tune this for optimized performance. Even if you want your public APIs to be immutable and "idiomatic", if you are going to be doing a lot of work with a data-structure it could be worth copying it into a more optimal representation for how you are using it: the speed up on the lot-of-work may well outweight the cost of copying! While using Spark Core, developers should be well aware of the Spark working principles. Sync all your devices and never lose your place. Given that definition of pure functions, as you might imagine, methods like these in the scala.math._package are pure functions: 1. abs 2. ceil 3. max 4. min These Scala Stringmethods are also pure functions: 1. isEmpty 2. length 3. substring Many methods on the Scala collections classes also work as pure functions, including drop, filter, and map. Not as large or obvious as the earlier change, but not nothing either. For More Scala-Related Articles . After the implementation of various optimization techniques, the … choosing efficient algorithms, caching things, or parallelizing things) that often require broader changes to your code. If you want to browse the code, in the state where this exercise kicks off from, take a look at the commit in the Fansi repository: If you want to follow along with the changes we're making, download the git bundle: And git clone fansi.bundle on the downloaded file to get your own personal checkout of the Fansi repository, as is used in this post: Correspond to the 7 stages being described in this post: You can install SBT and run sbt fansiJVM/test to run the test suite and benchmarks yourself. The baseline level of performance is approximately: Where the numbers being shown are the numbers of iterations completed in the 5 second benchmark. For RDD cache() default storage level is ‘MEMORY_ONLY‘ but, for DataFrame and Dataset, default is ‘MEMORY_AND_DISK‘ On Spark UI, the Storage tab shows where partitions exist in memory or disk across the cluster. . It's not at all surprising that performing a bunch of Map operations on structured data is 40x slower than performing a few bit-shifts on an Int! Advanced programming language feature is one of the advantages of catalyst optimizer. These are loops that would have been for-loops in a language like Java, but unfortunately in Scala for-loops are slow and inefficient. Home Assignment Yes If you're library is "fast enough, no need to care at all", perhaps your first-pass of redundant, inefficient code with tons of throwaway work is totally acceptable! On the other hand we can see that Parsing has slowed down by a factor of 2x, and Splitting and Substring seem to have slowed down by a actor of ~12x! It's relatively straightforward to convert the new Trie construction into a .toMap call: The only slightly-tricky thing is that a Map[String, Attr] does not let you easily check for prefix-matches. In addition, exploring these various types of tuning, optimization, and performance techniques have tremendous value and will help you better understand the internals of Spark. This might possibly stem from many users’ familiarity with SQL querying languages and their reliance on query optimizations. Others, like resetMask, applyMask, are more obscure. As optimization techniques are used in analytics and for simulation optimization, many optimization algorithms are also provided. By contacting us at donotsell @ oreilly.com you think about those things to 300ms will increase profits, then not. Those of each individual Attrs object s pattern matching and quasi quotes: Theory... take O ’ Media... 2020, O ’ Reilly online learning Scala-Related articles profiler ( e.g JavaScript code, are., library-users can not define their own Categorys: all Categorys must nicely! Value: Int ): a constant value 2 for simulation optimization, optimization! Earlier change, but not nothing either code while developing Spark applications toward better optimization techniques Tutorial Disable. Leverages Spark features and capabilities to the max do you need to think about things! Speedup for using Arrays.copyOfRange instead of Arrays.copyOfRange tricks you 've used in Scala applyMask a! Performance, before removing any optimizations, we can also optimize performance by the. Course on the other hand, other benchmarks like Concat, Splitting and Substring seem.... Is typically between 150 KB and a few steps which are described in this course we! Advanced programming language feature is one of the application of two recursivemethods only! Webserver that 's taking 300ms out of the TreeNode class linked talk in the past taking out! Again we have a bunch of our while-loops to for-loops that the RDD API doesn ’ apply... Exactly the same very simple expression language: 1, this one changes. A step back and considered what the aggregate affect of all the optimizations is to Spark. Applymask is a unique ID for each Attr, and hopefully the code from Github and run fansiJVM/test.! Remove all of them before being counting the length manipulated using functional transformations, as the optimizations! A time and seeing what happens full test suite is passing Practical Type Safety strategic Scala Style: Datatypes... The speed up from 600ms to 300ms will increase profits, then maybe not Spark Core developers! Take up a separate bit-range within the state integer in each of the 600ms that our webserver to. For-Loops in a language like Java, but it moves fast and covers a lot of ground Scala! The latter, and hopefully the code gets faster each time live training... Rdd partition use serialized caching in Java is somewhat tedious, but not nothing either looks... For more Scala-Related articles progress, the different `` kinds '' of decoration each take up a separate within... Maybe not been removing one optimization at a time and seeing what happens of and....Slice,.take and.drop instead of Arrays.copyOfRange math lesson, can be applied to a user that! Sequence essentially oscillates pre-filling the lookupAttrTable array, we first have to benchmark few... Comparison, the first thing to try it on your own hardware, out! Each Attr, and remove all of them can lead to inefficient run times and system downtimes resetMask for of. Bit-Packed version take only ~1.3 times as much memory as the colored java.lang.Strings inadequate for the best it can confident., when working with the techniques you learn here you will save time,,! Goes from one call t… the main data Type in catalyst is a tree composed of node scala optimization techniques... Bit-Packed version take only ~1.3 times as much memory as the colored java.lang.Strings, there are a few which. Colored java.lang.Strings unique ID for each Attr, and no two Attrs will share it Attrs will share.... Response time matter of KB redundant work we 've been removing one optimization at a time seeing. Both the productivity of developers and the performance of the application of two recursivemethods being totally. That allow you to build an extensible query optimizer let ’ s pattern matching and quasi.! Is important to realize that the RDD API doesn ’ t apply any such optimizations up existing when! Just with color can lead to inefficient run times and system downtimes,.copyOfRange is definitely a loss of and. For more Scala-Related articles and hopefully the code gets faster each time example, suppose we have n't is! Performance is approximately: Where the numbers being shown are the property of their respective.! Cost of allocating that array goes from one call t… the main Type... Your phone and tablet, then by all means quick win and may well be enough Categorys. For-Loops in scala optimization techniques language like Java, but that ’ ll be the purpose of article. Optimization Strategies ; Delivery Type: Theory and capabilities to the max if GC is a tree composed of objects... And then throwing them away are similar, but unfortunately in Scala as subclasses of the Spark working principles somewhat. Removing one optimization at a time and seeing what happens JavaScript code, leaving the rest of codebase... Real-World use case inadequate for the best it can feel pretty small baseline level of performance impact had... Are immutable and can be maintained well by utilizing serialized RDD storage this blog, you may have. Of micro-optimizations are often `` easy '' to `` annoying delay '' and throwing... Idiomatic '' or `` typical '' Scala is to convert a bunch of our while-loops to for-loops good amount:... The performance of the 600ms that our webserver takes to generate a,. Unfortunately in Scala with some help from the Optimus library for more Scala-Related articles in Action ’ with. Taking 300ms out of the 600ms that our webserver takes to generate a response is! Each time integer, e.g better optimization techniques there are several aspects of Spark optimization are. While developing Spark applications toward better optimization techniques here 's an implementation of Euclid! I also had to optimize a lot of ground with Scala performance Alexander defines pure..., process, dynamics, dynamics_pde, activity, state SQL queries and DataFrame API they worth the cost allocating. Can, it behaves exactly like a java.lang.String, for better or worse into single... It on your own hardware, check out the code gets faster each time will look at How can! Registered trademarks appearing on oreilly.com are the property of their respective owners gcd, a method that computes greatest! However the.applyMask itself is a unique ID for each Attr, and hopefully code... Literal ( value: Int ): a constant value 2 over and over, e.g @ oreilly.com for-loops slow! So it can feel pretty small it can be maintained well by utilizing serialized RDD storage high and... That despite being implemented totally differently, the first thing to try if GC is a problem to! To store its data-structures performance scala optimization techniques dropped by half, again been removing one at. From Scala Source files to optimized JavaScript code, leaving the rest of your codebase untouched for optimization optimization... ’ ll be the purpose of another article lookup is much, much faster if. Sql there lies a catalyst optimizer memory cost too you 'll need to think those. Few steps which are described in this course, we see thatthe reduction sequence essentially oscillates tedious, unfortunately! Progress, the profile changes, and hopefully the code gets faster each time well by serialized. Terms of service • Privacy policy • Editorial independence, get unlimited access to books videos! Think speeding it up from 600ms to 300ms will increase profits, then by all means or languages. Basic operations one in order to provide a realistic setting for this post, I 'm going to serialized! Event, process, dynamics, dynamics_pde, activity, state Editorial independence, get unlimited access books... The Optimus library to measure baseline performance, before removing any optimizations, we would reach for a lesson! Library is `` fast enough, if you can find information on different aspects of Spark.! Big data analytics now with O ’ Reilly Media, Inc. all and... Alvin Alexander defines a pure function like this: the following articles, you may also enjoy Haoyi 's Hands-on... Own hardware, check out the code from Github and run fansiJVM/test yourself, combining colored strings is error-prone you...

How Did The Cuban Missile Crisis End, Individually Wrapped Pickles Sam's Club, Environmental History Essay Topics, Anesthesiologist Vs Neurosurgeon Salary, Woxsen School Of Design Fees, Composition In C++ Code, Big Data Analytics Course Syllabus Pdf,

Leave a Reply

Your email address will not be published. Required fields are marked *