Measuring cloud performance just got easier and better

In February 2015, Google Cloud Platform and 30+ industry leaders and researchers launched PerfKit Benchmarker (PKB). PKB is an open source cloud benchmarking tool with more than 500 contributors from across the industry, including major cloud providers, hardware vendors and academia.

Today we're proud to announce our version 1.0 release. PKB supports 9 cloud providers, including AliCloud, Amazon Web Services, CloudStack, DigitalOcean, Google Cloud Platform, Kubernetes, Microsoft Azure, OpenStack, Rackspace, as well as the machine under your desk or in your datacenter. It fully automates 26 benchmarks covering compute, network and storage primitives, common applications like Tomcat and Cassandra, as well as cloud-specific services like object storage and managed MySQL. It also offers popular benchmarks such as EPFL EcoCloud Web Search, and EPFL EcoCloud Web Serving.

Since we first released PKB, we've seen strong engagement from researchers, industry partners and universities, making it a real community effort. PKB is being used today to measure performance across public clouds, bare-metal hardware and hardware simulations.

Fig. 1 PerfKit Benchmarker architecture

We're now declaring PerfKit Benchmarker V1 because the community believes we have the right set of benchmarks to cover the most common usage scenarios, the framework provides the right abstractions making it easy to extend and maintain and we've achieved the right balance between variance and runtime. PKB will continue to evolve and improve, covering new workloads and scenarios to keep pace with the ever changing cloud development design patterns.

Javier Picorel, a researcher from EPFL EcoCloud explained here why CloudSuite chose to integrate with PKB:

“Cloud computing has become the dominant computing platform for delivering scalable online services to a global user base all over the world. The constant emergence of new services, growing user bases and data deluge result in ever-increasing computational requirements for the service providers. Popular online services, such as web search, social networks and video streaming, are hosted by private or public cloud providers in large cloud-server systems, which comprise thousands of servers. Since its inception, CloudSuite has emerged as a popular suite of benchmarks, both in industry and among academics, to benchmark the performance of cloud servers. CloudSuite facilitates research in the field of servers specialized for cloud workloads (e.g., Scale-out Processors) and the development of real products in the server industry (e.g., Cavium Thunderx Processor). 

We believe that PerfKit Benchmarker (PKB) is a step towards the standardization of cloud benchmarking. In essence, we envision PKB as the “SPEC for cloud-server systems.” Naturally, our goals match with PKB's and the strong consortium put together by Google convinced us to team up. On the technical side, we are excited about the standard APIs that PKB provides, enabling the automated deployment of our benchmarks into the most well-known cloud-server providers. We believe that PKB has all the potential to be established as the de-facto standard for cloud benchmarking. Therefore, we expect it to grab the attention of cloud providers, academics and industry, while integrating more and the most recent online services.”

Carlos Torres is a Performance Engineer at Rackspace. At Rackspace, Carlos and his team help other developers performance test their products. They identify critical performance scenarios, develop benchmarks and tools to facilitate the execution and analysis of those scenarios, and provide guidance to other teams to develop their performance tests. Here's what he said:

“There are two main cases where I use PKB. One is to provide data for comparative analysis of hardware/software configurations to understand their performance characteristics, and the other is for measuring and tracking performance across software releases. PKB has brought me multiple benefits, but if I had to choose three, I'd say, speed, reproducibility and flexibility.

Speed: Before PKB, configuring and executing a complex benchmark that made use of a multi-node distributed system, such as a 9-node Hadoop cluster with HDFS, took hours of tedious setup, scripting and validation. Maintenance of those scripts, and knowing the current best practices for deploying such systems was a nightmare. Once you executed a benchmark, gathering the data from the tests usually involved manually executing scripts to scrape, parse and copy the data from multiple machines. Now, with PKB, it is very easy to execute, not one, but even multiple of these benchmarks, against every major cloud, usually with just one command. I can rely on the community's expertise, and for the most part, trust the configurations provided with each of the benchmarks. Finally, PKB makes is really easy to consume the data gathered from the tests, since it produces JSON output of the results. 

Reproducibility: Just like in science, reproducibility is a very important aspect of performance engineering. To confirm that either a bottleneck exists, or that it has been fixed, it is necessary to reliably reproduce a workload. Previous to PKB, it was tedious to keep track of all the software versions and configuration settings needed to replicate a benchmark, which sometimes were not documented and hence forgotten. This made reproducibility hard, and error prone. By using the same PKB version, with a single command, I can easily recreate complex benchmarks, and know that I'm executing the same set of software since PKB explicitly tracks versions of the applications, benchmarks and tools used. Also by just sharing the command I used for a test, other users can recreate the same test in no time. 

Flexibility: One of the best features of PKB is the ability execute the same benchmarks across different cloud providers, machine types and compatible operating systems images. While PKB ships with great defaults for most benchmarks, it makes it very easy to execute the benchmarks using custom settings, using commands switches or configuration files that a benchmark might optionally accept. PKB doesn't just make executing benchmarks easy, but contributing new benchmarks is simple as well. It provides a set of libraries that benchmark writers can use to write, for the most part, OS-agnostic benchmark installations.“

Marcin Karkocha and Mateusz Blaszkowski from Intel are working in the software-defined infrastructure (SDI) business to make reference implementations of private clouds.

“We try to determine how specific cloud configuration options impact on workloads running inside the instances. Based on this, we want to create reference architectures for different clouds. We also run Perfkit benchmarks in order to compare and calculate capabilities of reference architectures from different providers. In our case, PKB is used in private cloud — because of that we have slightly different requirements and problems to solve. We do not focus on comparing public cloud offerings. Instead we try to find out what is the most efficient HW/SW configuration for a specific cloud.

PKB as a framework gives us a possibility to create new plugins for providers and benchmarks. Thanks to this, we are able to easily build a custom benchmarking solution which meets most of our requirements.”

Daniel Sanchez and Assistant Professor at MIT EECS told us:

“We are using PKB to investigate new multicore systems. In particular, we are designing new hardware and software techniques that allow servers to provide low, predictable response latencies efficiently. 

PerfKit has made it much easier for us to simulate new hardware techniques on a broad array of cloud computing benchmarks. This is crucial for our work, because traditional benchmark suites are more focused on batch applications, which have quite different needs from cloud computing workloads.“

Our goal has always been to help customers make sense of the various cloud products and providers in a simple, transparent way. We want to be innovative, accountable and inclusive in our approach. We're happy to see this effort being welcomed by the industry and academia, and we welcome new partners and feedback.

We invite you to try PerfKit Benchmarker V1 to measure the cloud, and to join the Open Source Benchmarking effort on GitHub.