ds-load is a new CLI that helps extract, transform, and load data from a source (typically an identity provider) into objects and relations. These objects and relationships can be imported into the Aserto directory and used for authorization decisions. Read on for more!

The problem

At its heart, authorization is the process of computing a decision (allow or deny access) based on a subject (user), permission, and the object that is being accessed.

But where does all of this data come from? You could retrieve it all up front, and then pass it into your authorizer microservice. But that means that every caller needs to be able to obtain all this data up-front - users and their attributes, resources and their attributes, and relationships between them.

This defeats the goal of externalizing authorization, which is achieving a separation of concerns between the application code and authorization logic. Instead of making every caller obtain user, resource, and relationship information separately, the authorization system should store this information in a central place.

The solution

The Topaz authorizer contains an embedded directory which provides a solution for this exact problem: a place to store user attributes, resource attributes, and relationship information. The caller passes in a JWT to represent the user, a permission (or rule) name, and the key for the resource.

Authorization becomes a simple and efficient process:

• load the user from the directory using the subject extracted from the JWT as the key
• optionally, load resource from the directory (assuming it contains attributes that are relevant for authorization)
• check whether the user has a relation (or permission) on the resource by walking the relationship graph between the subject and the object

All of these actions can be done quickly and efficiently, given the indexed nature of the directory, and the efficient graph-walking algorithms it uses.

Complication

But how do we get all of this data into the directory? And how do we keep it in sync with the source? For relationships, the answer is typically that the directory is the source of truth for that data. But users and groups tend to be managed by your identity provider (Auth0, Okta, Cognito, etc). So how do we import users and groups into the directory?

ds-load to the rescue!

The ds-load CLI was built for exactly this purpose. It defines an extract-load-transform (ETL) pipeline for getting data from a source (typically an IDP such as Azure Active Directory, Cognito, Google Workspace, Auth0, or Okta) through a transformation, and into the Topaz (or Aserto) directory.

ds-load can pipeline all three actions to extract, transform, and load data from your IDP, or you can separate them into distinct stages.

Extract

ds-load comes with a set of plugins - one for each identity provider. It’s also pretty easy to write new ones - refer to the docs. This post focuses on the official plugins.

Each plugin supports a fetch command that obtains data from the source in its native format. For example, for Auth0, we can fetch users and roles using the following command:

ds-load auth0 fetch --domain=STRING --client-id=STRING --client-secret=STRING --roles >auth0.json

Transform

Each plugin comes with a default transformation, which will transform the data into objects and relations that conform to the Topaz directory format. The default transform for Auth0 will create user objects from Auth0 users, group objects from Auth0 roles, identity objects for each Auth0 identity, and relationships between all of these.

ds-load -p auth0 transform <auth0.json >auth0-transformed.json

Transforms are written as go templates, which gives you complete control over how the data is transformed. To learn about custom transformations, check out the docs.

Load

Finally, data in the directory format (a JSON file consisting of an array of objects followed by an array of relations) can be published into a Topaz directory, or any plug-compatible directory (such as the Aserto Directory):

ds-load publish <auth0-transformed.json

Conclusion

ds-load is a powerful tool that simplifies the process of extracting, transforming, and loading data from upstream sources such as an identity provider into the Topaz directory.

The ds-load CLI replaces the aserto-idp CLI, which provided a far more limited feature set for the Aserto v1 directory, and is now deprecated.

To check out the full feature set, head to the ds-load documentation!

We’re here to help!

If you need any help, find us on the Aserto community Slack, and we’ll be glad to answer any questions, help you write a custom transform, or even create a new plugin.

Happy hacking!