Comparing AWS Lambda Runtime Performance across Go, .Net Core 2.0, Node.js, Java and Python
This article deep dives into AWS Lambda performance metrics and was originally presented during Sydney Lambda Meetup January 2018.
AWS has released support for both Net Core 2.0 and Go 1.x on the 15 January. As an engineer who maintained serverless-golang and participated in a rather interesting Lambda dotnet performance discussion, I was very curious to see how the new runtimes compares in terms of performance. So I did a series of tests using Serverless Framework and obtained some very interesting results.
Setup
I based my tests on the 2017 lambda runtime performance comparisons by Yan Cui. For consistency, I forked his codebase and added some extra templates to test:
- aws-go: my go template merged as part of PR 4669 in Serverless Framework 1.26.0. Static binary built using go1.10 and -ldflags='-w -s'
- aws-go-small: same as the above, except using UPX to further compress the static binary.
- python-go: a once-off test using the now deprecated python shim
- aws-csharp2: csharp dotnet 2.0 template from upcoming PR 4656.
- aws-fsharp2: template proposed in my PR 4688 which will be released in 1.27.0
The code is available here. The Lambdas are configured with the default 1024MB memory set by Serverless Framework and deployed to North Virginia (us-east-1). Upon deployment there are 12 lambdas in total.
I created a script to call artillery.io to load test all 12 APIs across a one hour period.
Just like the original 2017 benchmark, we are ignoring the initial cold start time and focusing only on the duration metric to compare runtime performance between the different languages.
Observation 1 - .Net Core 2.0 significantly outperforms
Both C# and F# on .Net Core 2.0 exceed all expectations and outperforms all other runtimes in average duration. From now on .Net Core Lambda users should consider .Net Core 2.0 as the default and upgrade all existing projects still using 1.0.
As you can see from the graphs below, both C# and F# on .Net Core 2.0 are consistently lower than the rest over a 1 hour period:
Observation 2 - Go performance is comparable to Java
Go performance is very similar to Java with slightly less variance. There are no significant performance gains in using UPX compression nor the Python Shim.
For the same 1 hour period as above, Go average duration is exactly the same as Java:
Observation 3 - Static vs Dynamic languages are equally consistent in performance
Based on my results I wasn’t able to see any significant variances between average and maximum duration across compiled (Java, .Net, Go) and dynamic languages (Python, Node.js).
Observation 4 - Go packages are bigger than Java
Go binaries compiled with -s and -w linker flags results in a bigger package size (2.3mb) than Java (2.0mb). By using upx it is possible to shrink the binary size but it doesn’t make too much difference.
C# has the smallest footprint out of all compiled languages with 201KB. And F# adds on about an extra 1MB of dependencies.
To sum up
With Go and .Net Core 2.0 support, AWS continues to lead the FaaS race as the most mature provider with the widest range of language support. I am pleased to see the .Net Core 2.0 tooling on the Lambda platform dramatically improving in performance and stability since 1.0.
Personally as a Go advocate, I do feel the Go runtime may need some work. Judging from the fact that .Net Core 2.0 can run up to 3 times faster than Go, I suspect there might be some under the hood optimisation going on at the AWS sandbox and container level. But I am also confident that Go performance will improve as the AWS Lambda platform further evolves.
The above results do not reflect actual production use cases and the differences in performances are ultimately quite subtle. However compared to last year, architects and engineers can have more confidence in choosing the right tooling from all available runtimes, including latest .Net Core 2.0. In order to tackle the problems at hand.
Accelerated Serverless and Compliance with Convoy Framework
We can help you get to the cloud in days. Our DevOps mature cloud adoption and rapid application development framework Convoy can accelerate the bootstrapping of secure, cloud native environments with CD pipeline for serverless and/or containerised architectures.
Using Convoy, an enterprise could be roll out new applications which both achieve compliance and drive competitive advantage within days.
The Ultimate Guide to Serverless
The world of serverless has evolved considerably in the last ten years, but it can be difficult knowing where to start.
In this white paper, we discuss how large enterprise organisations can go on the serverless journey. Download it now to find out more!
