AWS, Lambda

Trying to make sense of serverless?

We can help!

Serverless computing enables you to concentrate on differentiating application code, rather than managing middleware, infrastructure and other commodities. To learn more, check out our whitepaper: Introduction to Serverless Computing with AWS Lambda.

Download the FREE guide →

This is part 1 of an Enterprise Serverless series focusing on serverless performance, cold starts, security, vendor lock-in concerns and production adoption within the enterprise. This article deep dives into AWS Lambda performance metrics and was originally presented during Sydney Lambda Meetup January 2018.

AWS has released support for both Net Core 2.0 and Go 1.x on the 15 January. As an engineer who maintained serverless-golang and participated in a rather interesting Lambda dotnet performance discussion, I was very curious to see how the new runtimes compares in terms of performance. So I did a series of tests using Serverless Framework and obtained some very interesting results.

Setup

I based my tests on the 2017 lambda runtime performance comparisons by Yan Cui. For consistency, I forked his codebase and added some extra templates to test:

  • aws-go: my go template merged as part of PR 4669 in Serverless Framework 1.26.0. Static binary built using go1.10 and -ldflags='-w -s'

  • aws-go-small: same as the above, except using UPX to further compress the static binary.

  • python-go: a once-off test using the now deprecated python shim

  • aws-csharp2: csharp dotnet 2.0 template from upcoming PR 4656.

  • aws-fsharp2: template proposed in my PR 4688 which will be released in 1.27.0

The code is available here. The Lambdas are configured with the default 1024MB memory set by Serverless Framework and deployed to North Virginia (us-east-1). Upon deployment there are 12 lambdas in total.


I created a script to call artillery.io to load test all 12 APIs across a one hour period:

        
         # api.txt is a list of API urls
for api in `cat apis.txt`
do
   echo "testing $api"
   artillery quick --duration 3600 --rate 10 --num 1 $api &
done
        
      

Just like the original 2017 benchmark, we are ignoring the initial cold start time and focusing only on the duration metric to compare runtime performance between the different languages.

Observation 1 - .Net Core 2.0 significantly outperforms

Both C# and F# on .Net Core 2.0 exceed all expectations and outperforms all other runtimes in average duration. From now on .Net Core Lambda users should consider .Net Core 2.0 as the default and upgrade all existing projects still using 1.0.

As you can see from the graphs below, both C# and F# on .Net Core 2.0 are consistently lower than the rest over a 1 hour period:

Observation 2 - Go performance is comparable to Java

Go performance is very similar to Java with slightly less variance. There are no significant performance gains in using UPX compression nor the Python Shim.

For the same 1 hour period as above, Go average duration is exactly the same as Java:

Observation 3 - Static vs Dynamic languages are equally consistent in performance

Based on my results I wasn’t able to see any significant variances between average and maximum duration across compiled (Java, .Net, Go) and dynamic languages (Python, Node.js).

Observation 4 - Go packages are bigger than Java

Go binaries compiled with -s and -w linker flags results in a bigger package size (2.3mb) than Java (2.0mb). By using upx it is possible to shrink the binary size but it doesn’t make too much difference.

C# has the smallest footprint out of all compiled languages with 201KB. And F# adds on about an extra 1MB of dependencies.

To sum up

With Go and .Net Core 2.0 support, AWS continues to lead the FaaS race as the most mature provider with the widest range of language support. I am pleased to see the .Net Core 2.0 tooling on the Lambda platform dramatically improving in performance and stability since 1.0. 

Personally as a Go advocate, I do feel the Go runtime may need some work. Judging from the fact that .Net Core 2.0 can run up to 3 times faster than Go, I suspect there might be some under the hood optimisation going on at the AWS sandbox and container level. But I am also confident that Go performance will improve as the AWS Lambda platform further evolves.

The above results do not reflect actual production use cases and the differences in performances are ultimately quite subtle. However compared to last year, architects and engineers can have more confidence in choosing the right tooling from all available runtimes, including latest .Net Core 2.0. In order to tackle the problems at hand.


Accelerated Serverless and Compliance with Convoy Framework

We can help you get to the cloud in days. Our DevOps mature cloud adoption and rapid application development framework Convoy can accelerate the bootstrapping of secure, cloud native environments with CD pipeline for serverless and/or containerised architectures.

 Using Convoy, an enterprise could be roll out new applications which both achieve compliance and drive competitive advantage within days.

 

Trying to make sense of serverless?

We can help!

Serverless computing enables you to concentrate on differentiating application code, rather than managing middleware, infrastructure and other commodities. To learn more, check out our whitepaper: Introduction to Serverless Computing with AWS Lambda.

Download the FREE guide →

x

SIGN UP TO OUR UPDATES

DevOps Insights Directly to Your Inbox!

Join thousands of your peers and subscribe to our best content, news, services and events.

  • Yun Zhi Lin

    VP of Engineering - APAC

    Yun heads up the Engineering capability for Contino APAC. He specialises in using data, serverless and higher order technologies to rapidly drive innovations, business strategy and measurable outcomes. In two decades, he has transformed multiple financial institutions, government, telcos and startups. Yun authored the popular AWS Lambda Performance Benchmarks, and is a veteran speaker at AWS Summit, ServerlessConf, numerous meetups and debates.

    More Articles by Yun