Issues with asynchronous IO in web applications

Building servers with non-blocking IO has been quite popular these days. Tests have shown that it does actually improve scalability of web applications. However, my experience show that it comes at a cost. In this post I am going to discuss some negative aspects of writing asynchronous code based on Scala’s Futures.

Stacktraces

Debugging exceptions in asynchronous programs is a pain. When issuing an asynchronous IO operation you provide a callback that should be executed when the operation returns. In most implementations, this callback might be executed on any thread (not necessarly the same thread that invoked the operation). Since call stack is local to the thread, the stacktrace that you get when handling an exception is not very informative. It will not trace back to the servlet so you may have hard time figuring out where what actually happened.

Thread-local variables

Some libraries use a mechanism called ThreadLocal  variables (available in Java and C#, in Scala known as DynamicVariable ). By definition, these libraries do not work well with asynchronous code, for the same reason that we get poor stacktraces.

I have already discussed one of such situations on my blog. Another one is Mapped Diagnostic Context from the Logback framework. MDC is a nice mechanism that allows you to attach additional information to your logs. Since the information is contextual, it will be available even to logs written from within external libraries. However, as one might expect, MDC is implemented with thread-local variables. Therefore, it doesn’t work well with Scala’s futures.

There is a way to get MDC with Futures working by writing a custom ExecutionContext  (Scala’s threadpool) that is aware of contextual data and propagates across threads.

Missed exceptions

Unless you are very careful, it is quite easy to not wait for a Future to complete but instead to fork execution into two branches. When an exception is thrown in a Future that nobody is waiting for, it will most likely just go unnoticed.

Above code will compile. However, saveToDb  will most likely be called before postData  returns since execution has been forked. Any exception thrown inside postData  will most likely be missed. The correct way to write the above code would be:

Caching

Caching gets more complicated in an asynchronous web application, unless the library you use for caching is designed to work with async code. One of the most common patterns in caching libraries is to let you provide a function that should be executed when a value in cache is missing. See the below example of Guava Cache:

If  doThingsTheHardWay returned a Future (was asynchronous) then you would have to block the thread and wait for the result. Mixing blocking and non-blocking code is generally discouraged and may lead to undesirable situations such as deadlocks.

Code readbility

Asynchronous code adds complexity. In Scala, you need to use all sorts of Future combinators such as flatMap , map  or Future.sequence  in order to get your code to compile. The issue is partially addressed by async/await  language extensions/macros (available for example in Scala and C#) but it can still make your code less readable and harder to reason about.

Using Automapper to improve performance of Entity Framework

Entity Framework is an ORM technology widely used in the .NET world. It’s very convenient to use and lets you forget about SQL… well, at least until you hit performance issues.

Looking at the web applications I worked on, database access usually turned out to be the first thing to improve when  optimizing application performance.

Navigation properties

The main goal of Entity Framework is to map an object graph to a relational database. Tables are mapped to classes. Relationships between tables are represented with navigation properties.

ef

The above example will be mapped to the following classes:

The highlighted lines declare navigation properties. Thanks to navigation properties, it’s very convenient to access details of Article’s Author. However, it comes at a cost. Imagine the following code in the view:

Assuming that ViewBag.Articles  is loaded with the below method, this code might turn out to be very slow.

Unfortuantely, it will fire a separate SQL query to the database server for each element in the Articles collection. This is highly suboptimal and might result in long loading times.

Lazy and eager loading

The reason behind this behaviour is the default setting of Entity Framework which tells it to load navigation properties on demand. This is called lazy loading.

One can easily overcome this problem by enabling eager loading:

Eager loading will cause EF to pre-load all Authors for all selected Articles (effectively, performing a join).

This might work for simple use cases. But imagine that Author has 50 columns and you are only interested in one of them. Or, Author is a superclass of a huge class hierarchy modelled as table-per-type. Then, the query built by EF would become unncessarly huge and it would result in transfering loads of unnecessary data.

Introducing DTO

One way to handle this situation is to introduce a new type which has all Article’s properties but additionally has some of the related Author’s properites:

Now we can perform projection in the query. We will get a much smaller query and much less data transfered over the wire:

Automapper

We improved performance, but now the code looks much worse – it involves manual mapping of properties which is in fact trivial to figure out. What’s more, we would need to change this code every time we add or remove a field in the Article class.

A library called Automapper comes to rescue. Automapper is a convention-based class mapping tool. Convention-based means that it relies on naming conventions of parameters. For example,  Author.FirstName  is automatically mapped to AuthorFirstName . Isn’t that cool?

You can find it on NuGet. Once you add it to your solution, you need to create Automapper configuration:

Here we declare that Article should be mapped to ArticleDto, meaning that every property of Article should be copied to the property of ArticleDto with the same name.

Now, we need to replace the huge manual projection with Automapper’s ProjectTo  call.

You need to add one more line:

And that’s it. You’ve just improved readability of your code and made it less fragile to changes.

Summary

Automapper is a very flexible tool. You don’t need to rely on naming convensions, you can easily declare your own mappings.

Additionally, we have just used just a specific part of Automapper – Queryble Extensions which work with ORMs. You can also use Automapper on regular collections or just on plain objects.

I believe the problem I highlighted here is just a symptom of a much broader issue of incompatibility of relational and object oriented worlds. Although Entity Framework tries to address the issue by allowing to choose between eager and lazy loading, I don’t think it is a good solution. Classes managed by EF being elements of a public API are a big problem. As a user of such interface you never know if a navigation property is loaded and whether accessing it will result in a DB query.

Therefore, I advocate the use of mapped DTOs. This approach reminds me slightly of an idea called Functional Relational Mapping adopted for example by the Slick framework for Scala. I believe it to be a great alternative to classic ORMs.

Some references:

Asynchronous programming in Scala vs C#

In one of my recent post I compared two different approaches that authors of Scala and C# chose to solve the same problem. This post is based on the same idea but the problem being solved is asynchronous programming.

What’s asynchronous programming?

Let me explain by giving you an example. If you have ever used a web framework you might have been wondering how it handles multiple concurrent requests from different users. The traditional approach is to spawn a new thread (or get one from a thread pool) for every request that comes in and release it once the request is served.

The problem with this solution is that whenever those threads perform IO operations (such as talking to a database) they simply block and wait for the operation to finish. Therefore, we end up wasting precious CPU time by allowing our threads to be blocked on IO.

Instead of blocking threads on IO operation we could use an asynchronous database API. Such API is non-blocking. However, running a database query using such an API requires you to provide a callback. Callback in this case would be a function that would be invoked once the result is available.

So, in the asynchronus model your thread serves the request, runs some computations and when it needs to call the database, it initiates the call and than switches to do some other, useful work. Some other thread will continue execution of your request when the database returns.

Asynchronous model example
Example of asynchronous processing in a web framework

Asynchronous programming in C#

The biggest pain of writing programs in the asynchronous model is the necessity of callbacks. Fortunately, in C# we have lambda functions which allow us to write callbacks with ease. However, even with lambdas we can end up with lot of nesting.

The key to asynchronous programming in C# is the Task class. Task represents a piece of work that can be either blocking or heavy on processor so it makes sense to run it asynchronously.

In the first line we create a task that fetches the Google main page. The task starts immedietely on a thread from a default, global thread pool. Therefore, the call itself is not blocking. On the second line we attach a callback which defines what should happen once the result is fetched.

As I said, it is easy to introduce nesting with callbacks. What if we wanted to visit Facebook but only if we succeeded fetching the Google page?

This code isn’t very readable. Also, if we wanted to visit more websites, we could end up with even more levels of nesting.

C# 5.0 introduced an excellent language feature that lets you write asynchronous code just as if it was synchronous: the async and await keywords. The above example can be rewritten as follows:

One caveat about async/await is that the method containing any await calls must itself be declared as async. Also, the return type of such a method must be a Task. Therefore, the asynchronous-ness always propagates upstream. This actually makes sense – otherwise you would need to synchronously wait for a task to finish at some point. Modern web frameworks such as ASP.NET MVC let you declare the methods that handle the incoming requests as asynchronous.

One more thing about C# tasks – with them executing stuff in parallel is incredibly easy.

Task.WhenAll creates a task that will be finished when all tasks from the provided array are finished.

Asynchronous programming in Scala

Let’s have a look at how Scala approaches the problem. One of the approaches to asynchronous programming is to use Futures. Future is a class that has very similiar semantics to C#’s Task. Unfortunately, there is no built-in asynchronous HTTP client in Scala, but let’s assume we’ve got one and it’s interface looks like this:

We can write code that looks very similiar to the C# example with flatMap:

Flatmap invoked on a future takes a callback that will be execute once the result of that future is available. Since that callback must return a Future itself, we must return an empty future (Future.successful) in the else branch of our if.

When fetching the Facebook page, we use map instead of flatMap because we don’t want to start another future inside the callback.

Again, the main issue with this code is that it is nested. Very similarly to how Scala handles nested null checks with Option monad, here we can again use the for-comprehension syntax to get rid of nesting!

As you might have expected, parallel processing is also supported with Futures:

An example of a web framework that supports asynchronous request handlers is Scalatra.

Conclusions

As you can see, C# and Scala approach asynchronous programming similliarly. What I find interesting here is how Scala handles callback nesting with the generic mechanism of for comprehension and C# introduces a separate language feature for that. This is exactly the same pattern as in Option monad vs null-conditional operator. To be honest, I find the async/await overall a bit more awesome – it really makes you feel as if you were writing synchronous code.

Update: as pointed out by Darren and Yann in comments, you can also do async/await in Scala thanks to this library. There is also a pending proposal to add it to the language that admits that it’s inspired by C#’s asyns/await syntax.

Conclusions after first four months of blogging

In this short post I name some random conclusions I had after the first four months of blogging. I hope this will be helpful for people who are considering starting their own programming blog (which I fully recommend to do!).

Total number of views: 16 000 

The number looks good to me although it gets interesting if we look at the distribution of views over different posts:

 

So, most of the views are due to my latest post, Scala’s Option monad versus null-conditional operator in C#. I submit most of my posts to Hacker News and this is also the main source of hits.
The conclusion here is that the title of the blog post really matters. I am yet to discover why this particular one caught attention but my suspicion is that with functional programming being a hot topic nowadays might be the reason.

Total number of posts: 10

This is much worse than what I aimed for (which is at least one post per week). The primary reason is lack of time since writing a longer piece is at least 2 hours for me.

What I plan to do about it is to do more short posts explaining solutions to some interesting problems I encounter at work or while working on side projects (such as Accessing request parameters from inside a Future in Scalatra).

My opinion on Blogger

I chose Blogger following the advice on one of other programming blogs. So far, I’m not totally happy with it and I kind of regret that I did not choose WordPress.

I once had a blog on WordPress for a while and what I liked there is that some of the traffic came from other WordPress users thanks to its Discover and Recommendations features. I thought a similiar thing will happen here with Google+ but it’s not happening at all.

Additionally, the choice of free templates is much poorer, the built-in editor is not very convenient and the statistics module is less fancy.

Update: I decided to move the blog to WordPress because of the reasons mentioned above.

Scala’s Option monad versus null-conditional operator in C#

Today I will talk about an awesome feature of C# 6.0. We will see how it can help us understand monads in Scala!

Null-conditional operator

Imagine we have a nested data model and want to call some method on a property nested deeply inside an object graph. Let’s assume that Article does not have to have an Author, the Author does not have to have an Address and the address does not have to have a City (for example this data can be missing from our database).

This is very unsafe code since we are at risk of NullReferenceException. We have to introduce some null checks in order to avoid the exception.

Yuck! So much boilerplater code to do a very simple thing. It’s really unreadable and confusing.

Fortunately, C# 6.0 introduces the null-conditional operator. The new operator denotes ?. and can be used instead of the regular . whenever it is possible that the value on the left can be null.

For example, the below piece can be read as “call ToUpper only if bob is not null; otherwise, just set bobUpper to null“.

Returning to our previous example, we can now safely write:

 

The Option type

As I explained in one of my previous posts, in Scala we avoid having null variables at all cost. However, we would still like to be able to somehow reflect the fact that a piece of data is optional. The Option[T] type can be used to explicitly mark a value as optional. For example, vale bob with type Option[String] means that bob can either hold a String value or nothing:

Therefore, we can easily model the situation from the previous example as follows:

Notice how, compared to C#, Scala forces us to explicitly declare which field is and which field is not optional.

Now, let’s look at how we could implement printing article’s author’s city in lower case:

This naive approach is not a big improvement when compared to the C# version. However, Scala lets us do this much better:

Although this version is not as short as the one with C#’s null-conditional operator, it’s important that we got rid of the boilerplate nested if statements. What remained is a much more readable piece of code. This is an example of the for-comprehension syntax together with the monadic aspect of the Option type.

The Option monad

Before I exaplain what exactly is going on in the above piece of code, let me talk more about methods of the Option type. Do you remember the map method of the List type? It took a function and applied it to every element of the list. Interestingly, Option does also have the map method. Think of Option as of a List that can have one (Some) or zero (None) elements. So, Option.map takes a function and if there is a value inside the Option, it applies the function to the value. If there is no value inside the Option, map will simply return None.

Now, can we somehow use it with our initial problem? Let’s see:

I think it looks slightly better than the nested if approach. The problem with this is that the type of cityLowerCase is Option[Option[Option[String]]]. The actual result is deeply nested. What we would prefer to have is an Option[String]. There is a method similiar to map which would give us exactly what we want – it’s called flatMap.

Option.flatMap takes a function that transforms an element inside the option to another option and returns the result of the transformation (which is a non-nested option). The equivalent for List is List.flatMap which takes a function that maps each element of the list to another list. At the end, it concatenates all of the returned lists.

The fact that Option[T] and List[T] have the flatMap means that they can be easily composed. In Scala, every type with the flatMap method is a monad! In other words, a monad is any generic type with a type parameter which can be composed with other instances of this type (using the flatMap method). Now, back to for-comprehension. The nice syntax which allows us to avoid nesting in code is actually nothing more than a syntactic sugar for flatMap and map. This:

…translates into this:

For comprehension works with any monad! Let’s look at an example with lists:

For each element in the first list we produce a list ranging from 1 to this element. At the end, we concatenate all of the resulting lists.

Conclusion

My main point here is to show that both C# and Scala introduce some language elements to deal with deep nesting. C# has null-conditional operators which deal with nesting null checks inside if statements. Scala has a much more generic mechanism which allows to avoid nesting with for-comprehension and flatMap. In the next post I will compare C#’s async keyword with Scala’s Future monad to show the similarities in how both languages approach the problem of nested code.

Accessing request parameters from inside a Future in Scalatra

Scalatra is an awesome, lightweight web framework for Scala. It’s perfect for building REST APIs. One of its less known features is support for asynchronous programming using Scala’s Futures.

By mixing in the FutureSupport trait one can easily make their servlet asynchronous. Once this trait is mixed-in into your servlet class, you can return Futures in your post and get handlers and Scalatra will automagically take care of them.

Recently I encountered a minor issue with Scalatra’s support for Futures – it is not possible to access params or request values from code inside a Future. The below code throws a NullPointerException.

Scalatra exposes access to contextual data such as the current user or request parameters via members such as params or request. These values are implemeted as DynamicVariables. Dynamic variables is Scala’s feature which allows a val to have different values in different scopes. The point is that DynamicVariable implementation is based on Java’s ThreadLocal. Therefore, when executing code in a Future you may not rely on these values since you might be on another thread!

An obvious solution to this problem is to retrieve request parameters before entering the Future:

However, this is not always a very convenient solution. I came up with the following workaround:

Firstly, we take a copy of the current request. Later, inside the Future we tell Scalatra to substitute the request dynamic variable’s value with our copy. Therefore, the call to params will use the correct request and there will be no error.

Update

As I recently learned, there is a much better way to solve this issue that is actually built into Scalatra. The way to go is using the AsyncResult  class. Our example would look like this:

AsyncResult  is an abstract class. We create an instance of anonymous type that extends it and overrides is  value. AsyncResult  takes copies of current request  and response  values when created and makes them available to code inside is .

You can find more information here.

 

Combining two objects in lodash

I’m a huge fan of the lodash library. It encourages functional programming style in JavaScript and as a result lets you write less error-prone and more readable code.

Some time ago I discovered the assign (extend) function which copies properties from target object(s) and assigns them to the target object. It also returns the target object. This is very handy whenever you want to combine properties from two objects into one object.

For example, given these two objects:

…instead of copying properties manually:

…one could write:

The result in c would be the same. However, with assign, object a will be modified. What I often wanted to do is to “merge” a and b into c but not modify neither a nor b.

Recently I learnt that the way to do this is the following:

With above code we copy properties of a and b into a new empty object and than return this new object. This is exactly what we wanted to achieve.

Scala for C# developers – part III

I’m back from a rather lenghty break and would like to continue the Scala for C# developers series. So far I have covered the syntax, the basics of OO in Scala and functions. In this post I will look at the Option type and pattern matching.

Issues with null references

If you have programmed in C# (or Java, or any other language that supports null references) you must already know the pain of NullReferenceException. This exception is thrown whenever you are expecting that a variable points to an actual object but in reality it does not point to anything. Therefore, calling a method on such reference would result in the exception.

There is a famous quote from Tony Hoare who introduced the concept of null references claiming that it was his billion-dollar mistake:

I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn’t resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.

The Option type

What does it mean when a NullReferenceException is thrown? As I said, it means that the CLR was expecting a reference to an object but found an empty refence and does not know what to do with it. In the majority of cases, it means that you as the programmer should have thought about it and check for the null reference before doing anything with it. Unfortunately, it would require some great discipline to keep track of all references that could become null and to take care of each and every one of them.

The Option type comes to rescue. The idea is to force the compiler to do the hard work for you.

Option[T] is an abstract type which has two subclasses: Some[T] and None. For example, a value of type Option[Int] represents an object that can, but does not have to hold some integer. If this Option is an instance of Some than the object has some value. If it’s None than it does not have any value.

So, None is like null except we explicitly declare that an object can be None by making it an Option. If we decide to use Option types in our project we must forget about null references completly. Therefore, whenever we expect a value of type T to be optional, we must declare it as Option[T]. Thanks to that, the compiler will forbid us from writing such code:

For this to compile, we must explicitly handle the case when the provided argument is undefined.

Such code, although lengthier, is much, much safer than traditional code which allows use of null references.

Of course, the key thing is to make sure that there is never a null inside Some value. However, this is easy to ensure as long as we decide not to use null references in the whole project.

Pattern matching

The above code snippet introduces some new syntax. The match
construct is the Scala syntax for pattern matching. It is a very powerful tool common in functional programming. You can think of it as a much more advanced switch statement which always returns a value as a whole.

In the above example, the value of textOpt is examined. It is an instance of Option type and we know that it has two subclasses. Therefore, there are two case branches. The first branch demonstrates how the value contained inside Some[T] can be extracted.

Pattern matching can be used with simple types:

Additionally, pattern matching works very well with case classes which we discussed in the previous post.

Why I like C#: feature-wise comparison with Java

Recently I was browsing Quora and was quite surprised to stumble upon this question: Java vs C#. Is Microsoft finally closing the gap? I decided to have a closer look and found more of similiar questions there.

Furthermore, at the place where I am currently working at, I am the only person with .NET background amongst mostly JVM people. We are all working on Scala projects and my colleagues are often surprised when I tell them that this or that Scala feature is also available in C#.

This makes me want to write a blog post about how cool a language C# is, especially when compared with Java. I want to underline that I’m speaking only about the language features and not about things like popularity, cross-platformness, ability to deploy easily, etc.

Generics

C# creators were in a great situation since they could learn from Java’s mistakes. They didn’t waste the opportunity and did the right thing. The main problem with Java’s generics is type erasure. The term means that the information about the type parameter of a generic type is not available at runtime. In simple words, this:

…becomes this:

Type erasure makes writing generic types more difficult and less clean. For example, sometimes generic methods have to explicitly take a Class object representing the type parameter (like here).

In C# this is not the case. You can easily access the type of the type parameter:

 

Lambdas, higher-order functions and LINQ

Not long ago I found this article on Hacker News. It discusses some of the new features of Java 8 such as lambdas, streams and functional interfaces. These things are called modern Java whereas in C# they have been available for quite a long time (not to mention that they have been available in Haskell or Ocaml for even longer).

While not everyone has to agree about superiority of functional over imperative programming, it’s hard to disagree that processing collections with higher-order functions (such as map/select or filter/where) is cleaner, less error-prone and much more readable than doing it with loops.

Even though Java has already adopted lambdas and higher-order functions, it seems that C# has better support for them. Examples?

  • In Java 8, you need to convert collections to Stream before calling map or filter
  • C# has built-in syntactic sugar for such opearations which makes such code even more readable and cleaner

 

Type inference

Type inference is a nice feature that allows you not to declare the type of a variable if it’s being initialized on the same line. While it’s not as great as in Scala or Haskell, it certainly lets you cut some boilerplate code. Java does also have some type inference but it is limited to generic methods.

With type inference, the below declaration:

…can be written as:

 

Asynchronous code

C# 5.0 introduced excellent support for asynchronous programming. The async and await keywords let you replace callback-style programming with code that looks exactly as if it were synchronous. It makes the code much cleaner and far easier to read.

The comparison with Java is especially striking if you look at pre-Java 8 code where in order to execute a piece of code asynchronously, you had to create an anounymous type with one method! Have a look at usage of the AsyncHttpClient library:

…and compare it with this C# code:

 

Value types

Value types is part of the reason why there is a C in C#. There are two kinds of types in C# – value types and reference types. Value types differ from reference types mainly in the assignment sementics. When you assign a reference to a new variable, this variable points to the same object. When you assign a value type to a new variable, the whole piece of memory holding the data in the type is copied. This is great for lightweight objects representing data. In some situations it might save you from writing the equals and hashCode operators. What’s more, value types cannot be null which makes them safer than reference types. Finally, value types make primitive types such as int or double more natural.

In Java, every type is a reference type.

Extension methods

Extension methods allow you to add functionality to an object (even if it had already been compiled). One of the cool uses of extension methods is providing a concrete method for an interface. Also, they allow better code reuse and makes it easier to write fluent APIs.

Example of an extension method:

 

C# 6.0 features

Finally, there are many great features introduced in C# 6.0. The language seems to be gravitating towards functional programming, which I think is a good idea, but most of them do not require the programmer to learn a new paradigm. To name some of the most exciting features of C# 6.0:

  • Expression bodied methods – syntax improvment which makes you write shorter code
  • Conditional null operator – which allows writing safer code (which feels like simplified Maybe monad)
  • Expression filters – convenient syntax for exception handling
  • Using static members – again, an improvment to make your code even shorter

 

Conclusion

I have named just a few of the language features of C# which I believe make it a superior language to Java. Obviously, there are many more things to look at when choosing a language than its features. However, I think it’s worth mentioning that thanks to Mono, Xamarin and Microsoft’s BizSpark program for startups, .NET became much more accessible to small companies and startups than it was a decade ago.

Scala for C# developers – part II

This is the second post in the series. Click here to see the previous part.

In the previous post I covered the basics of Scala syntax as well as some comparison of OOP in Scala and C#. Today, I will focus on lambdas and higher-order functions.

Functions as function parameters

You are most likely familiar with lambda expessions in C#. Lambda expression is simply an anonymous function. Lambdas are useful when you want to pass a piece of code as a parameter to some other function. This concept is actually one of the cornerstones of functional programming.

One great example of how useful lambdas are operations on collections. The following piece of code takes a list of integeres, filters out odd numbers and multiplies the remaining numbers by 5.

In Scala, it would look surprisingly similiar:

Scala uses more traditional FP names for map and filter but apart from this, the code looks very similiar. In Scala, we can make it a bit tighter (and less readable):

As you can see, Scala allows you to use anonymous parameters inside anonymous functions. However, be careful when using the underscore notation. The (_ * 5) + _ expression does not translate into x => (x * 5) + x. Instead, the second underscore is assumed to be the second anonymous parameter of the lambda, therefore meaning this: (x, y) => (x * 5) + y.

Returning functions

C# not only allows to have functions which take functions as parameters but also functions that return other functions. In the following piece, the GetMultiplier function takes a single integer and returns a function that can multiply it by any other integer.

Let’s see how would it look in Scala:

Again, it looks fairly similiar. The Function1[Int, Int] has the same semantics as Func%lt;int, int> – it represents a one-argument function that takes an integer and returns an integer. Interestingly, in Scala Function1[Int, Int] can be denoted as Int => Int.

We can go one step further and rewrite the above function as:

This certainly looks odd – our function now has two parameter lists! It is Scala’s special syntax for functions returning functions. You can pass one integer to getMultiplier and what you get is a partially applied function. What is the type of getMultiplier now? It’s Int => (Int => Int) which can also be written simply as Int => Int => Int. This technique is called currying. The idea of currying is that a function with multiple parameters can be treated as a function that takes the first parameter and returns a function that takes a second parameters which returns a function… and so on.