Storage Transform Plugin

Overview

The StorageTransformPlugin serves as a transformer of field values, as it targets only certain field type.

What Is It Used For?

The plugin is used to transform a certain field value to another value. For example, the time in format 15:45:59 is not usable when filtering, so we transform it to seconds and store it as a number.

When Is the Plugin Executed?

Storing the Data

When storing the data (create, update and createFrom) the plugin’s toStorage is executed just after the onBefore lifecycle events and before actual call to the storage operations.

storing the data flow

(click to enlarge)

Fetching the Data

When fetching the entries the plugin’s fromStorage is executed in the GraphQL resolvers, just before the output of the field value. We decided to have it like this because of possible unnecessary calls to fromStorage if we executed it just after fetching the data from the database.

fetching the data flow

(click to enlarge)

Why Execute fromStorage Just Before the Output?

For example, let’s imagine there is a entry field with a few megabytes of data in it - we stored it to S3 bucket. It is ok to fetch the data and output it, if the field was requested via the GraphQL. If the field was not requested, there is really no point in fetching remote data if it is not going to be used at all.

Examples

Time Transform

If you have a time field type, the input value of that field is in form of 15:45:59. That value is not comparable to another value when filtering in the database (or Elasticsearch in our case).

StorageTransformPlugin helps in this case by transforming the 15:45:59 into 56759, which are the seconds of the day, before we store the value into the database.

Here is how the time transform plugin should look like:

new StorageTransformPlugin({
  fieldType: "time",
  fromStorage: async ({ value }) => {
    const hours = Math.floor(value / 3600);
    const secondsAfterHours = value - hours * 3600;
    const minutes = secondsAfterHours > 0 ? Math.floor(secondsAfterHours / 60) : 0;
    const seconds = secondsAfterHours - minutes * 60;

    return [hours, minutes, seconds].map(value => String(value).padStart(2, "0")).join(":");
  },
  toStorage: async ({ value }) => {
    const [hours, minutes, seconds] = value.split(":").map(Number);

    return hours * 3600 + minutes * 60 + seconds;
  }
});

Text Transform

If you have a extremelyLargeText field type, which can get really large (couple of megabytes). You might want to compress it or not store it into the database at all (DynamoDB for example, as it has 400 kilobytes record limit).

Compressing the Value

The StorageTransformPlugin helps in this case by compressing the field value.

Here is how extremelyLargeText transform plugin should look like:

new StorageTransformPlugin({
  fieldType: "extremelyLargeText",
  fromStorage: async ({ value }) => {
    return decompress(value);
  },
  toStorage: async ({ value }) => {
    return compress(value);
  }
});

Of course, compress and decompress methods must be built as well. Or use some library for that.

Move to Other Storage System

If compression is not enough as the field value is extremely large, you can even move the value to some other storage system.

Here is an example for that:

new StorageTransformPlugin({
  fieldType: "extremelyLargeText",
  fromStorage: async ({ value: identifier }) => {
    return await fetchFromOtherStorageSystem(identifier);
  },
  toStorage: async ({ value, field, model }) => {
    const identifier = `${model.modelId}-${field.fieldId}-${randomlyGeneratedString()}`;
    await copyToOtherStorageSystem(identifier, value);
    return identifier;
  }
});

Of course, copyToOtherStorageSystem and fetchFromOtherStorageSystem are your own methods to copy and fetch the field value from other storage system.

Registering the Plugin

For the plugin to take effect you must register it in the system. File to register it in is api/code/headlessCMS/src/index.ts.

Note that if you register two plugins for the same field type, system will use the last one.

const handler = createHandler({
  plugins: [
    // ... other plugins
    new StorageTransformPlugin()
  ]
});

Conclusion

The StorageTransformPlugin makes it possible for you manage the value of the field, and where is it stored in case of storing to another storage system, so it is searchable.