We cordially invite the zk SNARK community to join us in creating a comprehensive benchmarking framework (zk-Harness) for zk SNARKs. As part of our efforts to further advance the technology and promote its widespread adoption, we have organized a ZKP-benchmark track in this ZKP/web3 Hackathon to bring together experts and enthusiasts from the community to collaborate and contribute to the establishment of a standardized benchmarking framework. This is a crucial step in the important mission to create a reference point for non-experts and experts alike on what zkSNARK scheme best suits their needs, and to also promote further research by identifying performance gaps. We believe that the collective efforts of the community will help to achieve this goal. Whether you are a researcher, developer, or simply passionate about zk SNARKs, we welcome your participation and contribution in this exciting initiative.

Some frameworks and tools in this track are developed as an initiative by the zk-Collective.

The zk-Harness is online at zk-bench.org!

Introduction

Figure 1 - zk-Harness Benchmarking Tool

zk-Harness

We have designed and developed zk-Harness as a general framework for benchmarking ZKPs, such that community members can easily contribute in a standardized fashion. The zk-Harness is designed for SNARK ZKP-frameworks that allow proving a circuit that describes a general computation. It is intended to be easily extensible - new ZKP-frameworks, hardware configurations and new measurement workload can be easily integrated. We specify a generic set of interfaces, such that benchmarks can be invoked through a configuration file (config.json), which produces a standardized output for a given benchmarking scenario (Arithmetic, Curve, Circuit, Recursion).

On a high level, zk-Harness takes as input a configuration file. The "Config Reader" reads the standardized config and invokes the ZKP framework as specified in the configuration file. You can find a description of the configuration file in the tutorials/config sub-folder of the GitHub repository. Each integrated ZKP framework exposes a set of functions that take as an input the standardized configuration parameters to execute the corresponding benchmarks. The output of benchmarking a given ZKP framework is a log file in csv format with standardized metrics. The log file is read by the "Log Analyzer", which compiles the logs into pandas dataframes that are used by the front-end and displayed on the public website. You can find the standardized logging format in the tutorials/logs sub-folder.

Currently, the zk-Harness supports gnark and circom to provide a set of examples of how a ZKP-framework can be integrated in the zk-Harness. We provide a set of simple benchmarks that can be found in the benchmarks directory.

There are several components remaining in the zk-Harness that you can contribute to! For example, there are several ZKP frameworks that are not yet included (plonky2, halo2, arkworks, jellyfish, zokrates, libsnark) which are required for a holistic comparison that benefits the community. Further, we currently only support benchmarks for a subset of the existing circuits available for the integrated ZKP frameworks. You can find a list of circuits that are available, but not yet integrated, here.

Program Task Description

There is a large and ever-increasing number of SNARK implementations. Although the theoretical complexity of the underlying proof systems is well understood, the concrete costs rely on a number of factors such as the efficiency of the field and curve implementations, the underlying proof techniques, and the computation model and its compatibility with the specific application. To elicit the concrete performance differences in different proof systems, we break the hackathon tasks into different categories to benchmark separately. We provide a high-level description in the following, you can find a detailed description here. Each of the following categories has designated tasks, which are further described in the detailed description. Beyond the tasks described, we also encourage participants to come up with self-selected tasks. The prices for each of the tasks will be announced soon.

Category 1: Mathematical Operations

All popular SNARKs operate over prime fields, which are basically integers modulo p, i.e., F_p. While some SNARKs are associated with a single field F_p, there are many SNARKs that rely on elliptic curve groups for security. For such SNARKs, the scalar field of the elliptic curve is F_p, and the base field is a different field F_q. Thus, the aim is to benchmark the field F_p, along with the field F_q and the elliptic curve group (if applicable).

Benchmarking F_p and F_q involves benchmarking the following operations:

Addition
Subtraction
Multiplication
(Modular) Exponentiation
Inverse Exponentiation

An elliptic curve is defined over a prime field of specific order (F_q). The elliptic curve group (E(F_q)) consists of the subgroup of points in the field that are on the curve, including a special point at infinity. While some SNARKs operate over elliptic curves without requiring pairings, others require pairings and therefore demand for pairing-friendly elliptic curves. The pairing operation takes an element from G₁ and an element from G₂ and computes an element in G_T. The elements of G_T are typically denoted by e(P, Q), where P is an element of G₁ and Q is an element of G₂. For efficiency, it is required that not only is the finite field arithmetic fast, but also the arithmetic in groups G₁ and G₂ as well as pairings are efficient. Therefore, we intend to benchmark the following operations over pairing-friendly elliptic curves:

Scalar Multiplication

in G for single elliptic curves
in G₁ and G₂ for pairing-friendly elliptic curves

Multi-Scalar Multiplication (MSM)

in G for single elliptic curves
in G₁ and G₂ for pairing-friendly elliptic curves

Pairings

for pairing-friendly elliptic curves

If you are unfamiliar with any of the above described concepts, you can find a good introduction here.

Category 2: Circuits

Many end-to-end applications require proving a specific cryptographic primitive, which requires the specification of said cryptographic primitive in a specific ZKP framework.

2.1 - Circuits for native field operations

These operations, namely, addition and multiplication in F_p, are supported by each SNARK library, and they are the most efficient to prove with a SNARK because arithmetic modulo F_p is the native computation model of a SNARK. This provides a good understanding of the efficiency of the core SNARK implementation.

2.2 - Circuits for non-native field operations

All computations we want to prove do not belong to arithmetic modulo p. For instance, Z_2^64 or uint64/int64 is a popular data type in traditional programming languages. Or, we might want to prove arithmetic on a different field, say Z_q. This usually happens when we want to verify elliptic-curve based cryptographic primitives. An example of this is supporting verification of ECDSA signatures. The native field of elliptic curve underlying the chosen SNARK typically differs from the base field of the secp256k1 curve (secp256k1 is not pairing friendly and hence does not present a suitable curve to instantiate SNARKs).

2.3 - Circuits for SNARK-optimized primitives

One of the challenges in the practically using SNARKs is their inefficiency with regard to traditional hash algorithms, like SHA-2, and traditional signature algorithms, such as ECDSA. They are fast when executed on a CPU, but prohibitively slow when used in a SNARK. As a result, the community has proposed several hash functions and signature algorithms that are SNARK-friendly, such as the following:

Poseidon Hash
Pedersen Hash
MIMC Hash
EdDSA

2.4 - Circuits for CPU-optimized primitives

Even though it would be beneficial to only rely on SNARK optimized primitives, practical applications often don’t allow for the usage of e.g. Poseidon hash functions or SNARK friendly signature schemes. For example, verifying ECDSA signatures in SNARKs is crucial when building e.g. zkBridge, however an implementation requires for non-native field arithmetic (see here), and therefore yields many constraints. Similarly, for building applications such as TLS Notary, one has to prove SHA-256 hash functions and AES-128 encryption which yields many constraints. Hence, we intend to benchmark the performance of the following cryptographic primitives and their circuit implementations in different ZKP-frameworks:

SHA-256
Blake2
ECDSA

For a detailed description on the configuration and log format expected for circuit benchmarks, we refer hackathon participants to the documentation section.

We integrated gnark to exemplify how to integrate libraries into the zk-Harness. You can find a description on how to run benchmarks for gnark here.

Category 3: Supporting new ZKP-frameworks

There are many implementations of SNARKs that we intend to include in the zk-Harness. Hence, we encourage hackathon participants of the benchmarking track to integrate novel libraries that are not yet supported in the initial version of zk-Harness to support a holistic comparison of heterogeneous SNARK implementations.

Choose a framework to integrate (e.g. arkworks / plonky2 / jellyfish)
Support the data loading of configuration files in the "Config Reader"
Configure the framework behavior based on the configuration file
Generate logs for a specified logging format (see logging formats here)
Integrate the logs in the frontend implementation
Create a pull request to integrate the implemented ZKP-framework in the zk-Harness and display the evaluation results on the public website.

You can find the detailed description on how to add a new library here. You can find the standardized, cross framework, log format which is consumed by the log analyzer here and the description of the generic config files here. To fully integrate a framework, you’ll need to adapt the config reader to invoke your benchmarking script and adapt the log analyzer to display your benchmarks on the webpage. The minimal integration of a new ZKP-framework in the zk-Harness should provide a “toy example” circuit.

Category 4: Recursion

Commonly, proof recursion can be achieved through the following approaches:

Recursion via succinct verification of SNARKs

Recursive verification of a SNARK can be achieved by essentially proving the correct verification of a SNARK. As such, there are several challenges in choosing the optimal curve and field (e.g. read about 2-chains and cycles of pairing-friendly elliptic curves here).

Recursion through accumulation/folding schemes

Instead of verifying SNARKs, accumulation schemes essentially “remember” the verification of previously encountered proofs in an accumulator value, such that the time complexity of the n-th recursion step is independent of the number of previously performed recursion steps.

In this task, you should benchmark common implementations of recursion in popular ZKP frameworks.

Category 5: zk-EVMs

In this task, you should benchmark the performance of current zk-EVM implementations

A zkEVM uses SNARKs to make validity proofs of the execution of Ethereum-like transactions. As such, the validity proof provided by a zkEVM enhances the scalability of Ethereum. Rather than re-executing transactions for validation of correct execution, only a short proof of correct execution has to be verified. Thus far, there are several approaches to zkEVMs (see here):

zkEVM with full Ethereum equivalence
zkEVM with full bytecode level equivalence
zkEVM with partial bytecode level equivalence
zkEVM with language level equivalence

NOTE - Not all zkEVM prover implementations are open-source and running a zkEVM prover can require large amounts of memory.

Prizes

A total of up to $5K will be awarded to the top three contributors

Detailed Description

You can find a detailed description of all tasks for the zk-Benchmarks Track here.

License

All accepted code submissions will be published in corresponding open-source GitHub repositories and licensed under the MIT license.