Open Policy Agent 2021, Year in Review

Published in

Open Policy Agent

10 min readJan 11, 2022

Although we set the bar high in 2020, 2021 turned out to be just as eventful as we anticipated — both for the Open Policy Agent (OPA) project and the world around us.

The year had barely started when OPA graduated in the Cloud Native Computing Foundation (CNCF). This achievement further proves that OPA is a mature, general purpose solution for policy enforcement and the “de facto” standard for authorization in the cloud-native environment. The graduation is a huge milestone, and an acknowledgement to everyone who worked tirelessly on the project since its inception. But just as much, it sparked a boom of interest for the project, bringing in a new wave of community members, code contributors, and production use cases. This is evident from the project’s 2021 usage metrics and numbers. Truly a great time to be involved in OPA!

Before we take a closer look at all the things that happened in and around OPA for the past year, let’s zoom out a bit for some context. What was 2021 like for tech communities in general?

After almost two years of conferences, meetups and other events being held online, it’s probably fair to say a sense of “online exhaustion” started to grow. With so much time of the working day being spent in online meetings and calls, trying to entice an audience to join even more online events — no matter how interesting or fun — became increasingly difficult. With large parts of the tech community getting vaccinated during the course of the summer and later, the return of in-person events thus felt like a big relief to many, and to some extent a return to “normal.” Whether this will last over 2022, or if we’ll see a return to online only events, remains to be seen. Either way, organizers of events will likely need to consider hybrid models for the foreseeable future, which, while certainly challenging, if done well also provides new opportunities in terms of geographical diversity, inclusion, and more.

In the security space, 2021 saw an increased interest in supply chain security, and it’s likely that trend continues over the next few years as our systems increasingly have come to rely on dependencies from all across the vast open source ecosystem. We also saw many tech teams, bloggers and conference talks discuss the concept of “DevSecOps, and what adding security — not as an afterthought but as an integral part of the application lifecycle — meant in terms of governance, team autonomy and agile workflows. While certainly coined before 2021, the concept of “zero trust” and architectures where security is considered in all parts of the stack, seemed to be the topic of a lot of discussions in the community over the last year. Having tools that offer a unified way of working with key security concepts like policy across diverse, heterogeneous domains and tech stacks will likely be more important than ever. The enormous increase of OPA downloads is not coincidental!

While not as easily measured in numbers, another exciting trend in the OPA ecosystem is the number of new organizations, corporations, startups and projects that use OPA not just as a component in their internal systems, but as an integral part of their own products and services — commercial and open source alike. As an emerging field in IT security, we’ve likely just experienced the start of a Policy-as-Code movement. What an exciting space to follow as it unfolds and develops over the next decade!

OPA 2021 in numbers

~ 85 million new downloads, almost tripled from last year (29 million)!
~ 1600 new GitHub stars, reaching 6000 in total just in time for 2022!
~ 1800 new Slack users
~ 702 commits
105 contributors
10 major versions released, 11 point releases
12000 VS Code OPA plug-in installs

Notable features and enhancements

Management

OPA saw a number of new methods for authenticating against remote services, like:

With these additions, OPA easily integrates with cloud providers like GCP and Azure using more idiomatic methods than previously supported.

The decision logger saw many improvements over the year, allowing truly large scale deployments where many thousands of decisions might be logged every minute.

Tooling

The opa eval subcommand saw support added for schemas (provided with the -s/ --schema flag). This allows a policy author to type check the input and data variables referenced in their Rego policies, catching simple mistakes like typos, or more serious type errors and references to properties that don’t exist. The schema support was one of the largest contributions to the project this year. Thanks @aavarghese and @vazirim!

Two new subcommands were added to OPA: opa refactor movecan now be used for package renaming use cases, and opa inspect can be used to quickly inspect the contents (including metadata) of a bundle.

The opa fmt subcommand can now exit with a non-zero status when a diff is detected (by passing — fail), and the opa test framework now supports “skippable” tests by prefixing a test name with todo_.

While it won’t be available until the first release in 2022, a new opa exec command has also been added to OPA, bridging the gap between running OPA as a server (i.e. opa run --server) and using OPA for one-off tasks using opa eval. This is particularly useful when using OPA in CI/CD contexts, where you might want to evaluate configuration or deployment resources without running OPA as a server, while still allowing policy and data to be fetched from remote endpoints, decisions to be logged to your control plane… and any other capabilities that running OPA with a config file allows.

Library

OPA now includes a new top-level package to support OPA integrations in Go programs: github.com/open-policy-agent/opa/sdk. Users that want to integrate OPA as a library in Go and expose features like bundles and decision logging should use this package. The package is controlled by specifying an OPA configuration file. Hot reloading is supported out-of-the-box. See the package docs for more details.

Experimental support for disk-based storage — as an alternative to in-memory storage — was made available to library users, with support in the server coming in early 2022.

Server

The server now supports a GET /v1/config endpoint that returns OPA’s active configuration. This API is useful if you need to debug the running configuration in an OPA configured via discovery.

Plugins can now be triggered manually within OPA. This feature allows users extending and customizing OPA to control exactly when operations like bundle downloads and decision log uploads occur. The built-in plugins now include a trigger configuration that can be set to manual or periodic (which is the default). When manual triggering is enabled, the plugins will not perform any periodic/background operations. Instead, the plugins will only execute when the Trigger API is invoked. Huge thanks to @gshively11 who both worked on this, and wrote about using OPA in serverless contexts in this blog.

Although yet to be released, the year ended with OpenTelemetry support landing in OPA. This makes it possible to emit spans to an OpenTelemetry collector via gRPC on both incoming and outgoing (i.e. http.send) calls in the server, and we’ll most certainly see a lot of interesting integrations and use cases leveraging this in the year to come. Thanks @rvalkenaers for all your hard work on this!

WebAssembly (Wasm)

The Wasmtime runtime is now the default runtime used by OPA to execute compiled policies.

The eval, test, bench and REPL subcommands now support a -t/--target flag to set the evaluation engine to use. The default engine is rego, but users can now select wasm to enable Wasm compilation and execution of policies (#2878).

The generated Wasm modules became much smaller: unused functions are now replaced by unreachable stubs, and the heavyweight runtime components related to regular expressions are excluded when none of the regex-related builtins are used.

A bunch of built-in functions saw native Wasm implementations added, like graph.reachable, json.filter, json.remove, object.get, object.remove, and object.union.

Performance

OPA got even faster in 2021! Among the largest optimization efforts was the implementation of an early exit strategy included for policy evaluation. Just like the name implies, this means the evaluator now will exit early when it can safely determine that continuing evaluation isn’t going to change the outcome of the evaluation. Examples of an early exit scenario include iteration over collections checking for a specific condition, or evaluating complete rules where an outcome has already been determined. Consider for example the following policy:

package earlyexitallow {
    is_admin
}allow {
    is_public_endpoint
}allow {
    input.path == ["users", input.username]
}

Should any of the conditions in the allow rule be true, the evaluator now exits immediately, as it is safe to do so. Another example, iterating over a list of ACLs, stopping as soon as all the conditions in the rule are met:

allow {
    acl := data.acls[_]
    acl.action == input.action
    acl.resource == input.resource
}

The decision logger saw a number of performance improvements, where the most impactful one is a new adaptive mode, where upload chunk sizes are calculated based on previous outputs. This allows the decision logger to encode significantly more decisions into each upload chunk.

Rego & built-in functions

Rego saw some major new features added in 2021, with the new in operator and print function likely the ones that stand out the most. The new in operator provides a quick, easy to read, way of determining membership in collections like arrays, objects and sets without iteration. This is particularly useful when checking that an item is not part of a collection, which previously required a separate helper rule:

deny["User must be admin"] {
    not "admin" in input.user.groups
}

Together with the previously existing some keyword, in may also be used as an alternative way of iterating over a collection:

ad_groups[group] {
    some group in input.user.groups
    startswith(group, "ad:")
}

Based on the yearly survey summary, better support for debugging Rego was highlighted as a topic where OPA users wanted to see improvements. While more investments are certain to follow in this space in 2022, the new print function has already proven to be tremendously useful for a debugging experience people have gotten used to from other languages, and naturally want to bring with them to Rego. While seemingly simple in nature, adding the print feature to a mature language like Rego required some special considerations, like how to handle undefined values, variable number of arguments, and more. While it took a few iterations to get right, the end result is a familiar way of quickly identifying what’s going on during policy evaluation. For more information about this new utility, check out the blog on the print function.

Eight new built-in functions were added to OPA this year, including functions for:

Math: ceil, floor, rand.intn
Date/time: time.diff
Cryptography: crypto.x509.parse_and_verify_certificates, crypto.x509.parse_rsa_private_key
Networking: net.lookup_ip_addr

Ecosystem and integrations

Gatekeeper

2021 was an intense year in the Gatekeeper project! The mutation feature graduated to beta, and a new method for validation using external data was introduced (currently in alpha). The new Gator CLI makes it possible to test constraint templates and constraints locally (i.e. without Kubernetes running), and makes for a very welcome addition to the tooling available around the project. Rather than denying requests, Gatekeeper policies may now also emit warnings as an enforcement action.

Other notable improvements include the ConstraintTemplate custom resource definition getting promoted to v1, Kubernetes v1.22+ compatibility, a new prefix-based matching option for namespaces and excluded namespaces, and new metrics reporting for mutation.

Conftest

2021 saw a lot of improvements to our favorite tool for testing configuration files. Three new file formats (.tfvars, .properties and .env) were included along with a new output format for GitHub, enabling violations to be printed inline in pull requests.

The ability to skip tests prefixed with todo_ — which was added to OPA earlier this year — also made its way into conftest. The new conftest fmt command now enables formatting of Rego policies, identical to that provided by opa fmt, for the unfortunate souls without an OPA binary installed on their systems.

On the distribution front, conftest got added to the core repository of the popular brew tool, meaning installation on Mac OS is now as simple as brew install conftest. Finally, the project saw large improvements made around reporting and tracing.

kube-mgmt

A rather nice surprise in 2021 was the increased activity seen in the kube-mgmt project. The project — which predates OPA Gatekeeper as a tool for using OPA in Kubernetes admission control scenarios — saw many improvements during the year, Much thanks to the new maintainer @eshepelyuk, but also from external contributors. A key strength of OPA is its versatility and general purpose nature, so it makes perfect sense to see many options and approaches to tackling problems.

Integrations

Some integrations that got added to the OPA ecosystem page include a plugin for the Apache APISIX gateway, documentation for integrating OPA with SPIRE/SPIFFE, data provisioning with the OPAL project, a Spring Security Reactive component, attestation verification for cosign, Python integrations for Bottle and Flask, PHP Symphony middleware, Google Calendar integration, a Clojure middleware, the Kubernetes security scanner Kubescape, and a tool for generating Sphinx docs from Rego.

Additionally, the Awesome OPA list got a lot of attention in the community, and saw many new integrations, blogs and articles added throughout the year.

Community

The community around OPA worked at an amazing pace in 2021! Despite many planned events getting either cancelled or moved online due to the pandemic, people in the OPA community managed to meet and share their experiences all over the world — in-person and online. We saw talks on OPA at pretty much all the major conferences, panel discussions around policy as code, meetups covering OPA in nearly all its popular applications, like infrastructure as code, Kubernetes admission control and of course, app authorization. 2021 also saw a couple of meetups entirely dedicated to OPA, and many new connections and friendships were formed in the community.

A new monthly newsletter was started in order to keep track of all the new projects, events and other happenings popping up in the community. If you haven’t already, make sure to check it out!

Finally, a new GitHub discussion board for OPA was launched, adding to the many channels available to the community for discussions, questions and resolving of issues around OPA.

Credits

The OPA community would not be what it is without all the fun, generous, smart and kind people involved. You all contributed to the success of OPA in 2021, whether you did so as a maintainer, contributor, integrator, adopter, user, by recommending OPA to a friend or colleague, or by helping someone out by answering their questions. With the dramatic increase of OPA installs this past year, it’s probably safe to guess that 2022 will be a year full of activity, in OPA as much as in the community. If you’re curious to know more about some of the things planned for OPA in 2022, the OPA roadmap might provide you some insights. The future is rarely predictable though, and I’m sure many of the most exciting things and events to come are yet to be discovered. Thank you all!