Figs: A Remote Configuration Service

Apr 11, 2023·5 min read

In the last few months I’ve been focused on building out more hybrid infrastructure for nendlabs and my home lab. One of the most pressing problems has been the need for remote configuration of systems and services. I’ve worked with similar systems in the past, usually built internally for their respective companies, so thought it would be a pleasant idea to take a stab at it. Its also a great excuse for me to play with some Rust.

Figs is a REST API for working with remote configuration objects. The interface resembles a key-value store where configurations are keyed by identifiers and map to an object containing some pre-defined settings.

Concepts

At its core, figs is a key-value store fronted by an API. With the intention to serve multiple applications and use cases, its important that enough flexibility is given to consumers while maintaining some sane defaults, and patterns. Frankly I’m still torn if this should be a Redis backed npm package, but where’s the learning in simplicity.

Namespaces

To support varying customers and use cases, a fundamental piece of the system are Namespaces. Namespaces are global and unique identifiers for consumers. Ideally these are applications, or domain spaces which then have many different configurations under them. For example my application hosted at lignes.dev could have a namespace matching its domain. Internally this namespace will be used to prefix all configurations under this domain.

Rather than be set by consumers, the Namespace for a given consumer will be derived from its Auth0 authentication claims. In the future, we may add other properties to authentication claims, such as limits for number of configurations and so on.

Configurations

With Namespaces provisioned, consumers can then use the exposed endpoints to manipulate configurations under their Namespace. Reusing our previous example, the lignes.dev consumer could create a global configuration which would apply to any readers. It could resemble the following snippet:

// sample key: nendlabs:figs:config:lignes.dev:global
{
  "og.description": "a generative art platform from nendlabs",
  "og.banner": "https://abs.lignes.dev/og",
}

This snippet could be useful for global settings that should apply to all instances or deployments. However figs allows consumers better target configurations with their own identifiers. The previous snippet could be slip into two to simulate an AB test.

// bucket: A
// sample key: lignes.dev:global:a
{
  "og.description": "a generative art platform from nendlabs",
  "og.banner": "https://abs.lignes.dev/og",
}

// bucket: B
// sample key: lignes.dev:global:b
{
  "og.description": "a generative art platform from nendlabs",
  "og.banner": "https://abs.lignes.dev/og",
}

Similarly this principle could be used to create configurations for specific end users, groups, and so on.

Design

The figs service is a series of lambda functions, written in Rust and deployed to Vercel. I chose Rust mostly because I was looking for a reason to try it, and after doing a few projects with it I’ve come to admire it for its reliability. Reliability can be misconstrued, so more definitively I like it because typically when I write things in Rust, they work the way I expect. This is great for things I want to codify and forget about, like figs.

The API is backed by Redis for persistent storage, and there were a few things I wanted to get right for the initial version:

Authentication

Authentication is handled by Auth0, using their machine-machine flow. This allows granular access for each consumer, as well as metadata associations with each consumer powered by their platform.

All requests to the platform must be authenticated, and setting up new consumers should be done from within Auth0.

Namespaces

Consumers onboarded to the system will be uniquely identified with Namespaces. Every consumer must be setup in Auth0 with the appropriate metadata. The authentication handling from within figs will extract the right namespace for the authenticated request.

For every Namespace, keys are deliberately formatted to leverage existing Redis operations. For example, any keys created for a consumer lignes.dev will resemble nendlabs:figs:config:lignes.dev:<configuration id> . This format makes operations on multiple keys straightforward for any given consumer.

Configurations

Configurations are the bread and butter of the system. Consumers can create arbitrary configurations based on application requirements. Some examples of this are shown above. Internally configurations are stored as Redis Hash Sets, keyed using the standard format. For example a consumer lignes.dev storing a configuration global will have the following key: nendlabs:figs:config:lignes.dev:global .

Consumers can also choose to segment their configurations or scope them using identifiers internal to their systems. For example nendlabs:figs:config:lignes.dev.<some user id>. All configurations are managed through the provided endpoints, and since all data is encrypted at rest we manually cannot view or edit them.

Endpoints

The interface of the API. It was important that I got this right and in a way that would scale if more functionality was needed. The core endpoints are as follows:

All requests must be authenticated and the respective Namespace will be derived from the authenticated session. Each endpoint is implemented as a Rust serverless function, sharing a lot of common functionality through shared modules.

Encryption

All keys and values are encrypted at rest but not in transit. This is partly because the Redis backend is provided by Upstash. They’re not awful or anything, but still an external vendor. In transit the service is hosted with the appropriate SSL certificates, but consumers can choose to encrypt their data as they choose.

Conclusion

This project has been a fun exercise digging into Rust, and getting back into designing and implementing pure backend services. There are some outstanding items on the roadmap but I’m excited to get started using figs for existing and upcoming nendlabs projects.