Introduction

Expression evaluation

Datadance at its core uses MozJexl expression language. We can use expression language to parse a given expression and evaluate it (1). For example 2+3 is parsed and evaluated to 5 by the Mozjexl. Additionally, a context can also be provided, where we can use it in our expressions as explained in the below example.

1. Mozjexl do not use eval() or any similar functions internally, instead it uses a grammar to parse expressions and then evaluates them using pre-defined operators and their behavior. Which makes it fast and safe to use.

Evaluating expressions using Mozjexl

const context = {
      "name": {
        "first": "Malory",
        "last": "Archer"
      },
      "exes": [
        "Nikolai Jakov",
        "Len Trexler",
        "Burt Reynolds"
      ],
      "lastEx": 2
}

Now I can perform the following :

mozjexl.eval("lastEx > 1 ? 'Moved on' : 'Stupid'", context).then(function(res) {
    console.log(res); // Output: Moved on
});

There are a lot of other expression languages out there, which you can use to perform similar expression evaluation.

Datadance takes advantage of this, by defining the transformations as a series of operations (1).

Below is a simple example how transforms are defined :

Note

Datadance uses JSON specification to define transform logic for transforming given input JSON to output JSON. The transformations are essentially a list of operations. Each operation in the list representing a JSON object with exactly one key-value pair.

The value of this key-value pair can be:

• A string representing a transformation expression.
• Another list of operations, where each operation also follows the same rule of having a JSON object with only one key-value pair.

This structure supports both simple and nested transformations, ensuring each operation is clearly defined with a single key.

• Input JSON

  ---

  {
    "name": {
      "first": "Malory",
      "last": "Archer"
    },
    "exes": [
      "Nikolai Jakov",
      "Len Trexler",
      "Burt Reynolds"
    ],
    "lastEx": 2
  }
  

• Transformed JSON

  ---

  {
    "name": {
      "first": "Malory",
      "last": "Archer"
    },
    "exes": [
      "Nikolai Jakov",
      "Len Trexler",
      "Burt Reynolds"
    ],
    "lastEx": 7,
    "original": 2,
    "modified": 7 
  }
  

• Transforms

  ---

  [
    {"lastEx" : "input.lastEx + 5"},
    {"original" : "input.lastEx"},
    {"modified" : "derived.lastEx"}
  ]
  

The expression input.lastEx evaluates to 2, which is taken from the context i.e Input JSON. Now input.lastEx + 5 evaluates to 7 and is stored in key lastEx.

But in the above example, input.lastEx evaluates to 2 but derived.lastEx evaluates to 7. To understand this, we need to learn about input & derived states.

Input & Derived states

In the above example we have used two different expressions to access the same item from the context.

1. input.lastEx : The input.<field> will fetch the value of lastEx from the Input state. Input state is immutable. For example, we can update the lastEx value multiple times in our transforms, but input.lastEx will always give the original value of lastEx from the input.
2. derived.lastEx : The derived.<field> will fetch the value of lastEx from the Derived state. Derived state is mutable. All the changes made, or new fields introduced through transforms (e.g. : original & modified) are updated in the derived state. So we can use derived state whenever we want to access the updated/latest values.

Initial derived state

Even before transforms are executed, the input state is copied to derived. So input and derived state contains the same context at the start of execution of transformations.

Nested transformations

You can also define deep-nested transformations, using the same principles explained above.

For example :

• Input JSON

  ---

  {
    "name": {
      "first": "Malory",
      "last": "Archer"
    },
    "exes": [
      "Nikolai Jakov",
      "Len Trexler",
      "Burt Reynolds"
    ],
    "lastEx": 2
  }
  

• Transformed JSON

  ---

  {
    "name": {
      "first": "Malory",
      "last": "Archer"
    },
    "exes": [
      "Nikolai Jakov",
      "Len Trexler",
      "Burt Reynolds"
    ],
    "lastEx": 2,
    "nameObject":{
      "name": "Malory Archer",
      "age": 2,
      "lastEx": {
        "name": "Burt Reynolds",
        "age" : 30
      }
    } 
  }
  

• Transforms

  ---

  [
    {"nameObject":[
      {"name" : "input.name.first + ' ' + input.name.last"},
      {"age": 35},
      {"lastEx": [
        {"name": "input.exes[input.lastEx]"},
        {"age": "derived.nameObject.age - 5"}
      ]}
    ]}
  ]
  

Playground

For convenience write transforms in YAML format in playground

In the playground, instead of writing the transforms in JSON format, you can simply write in YAML format, and click on Copy parsed transforms button to convert the YAML to transforms in the JSON specification format and copy it to clipboard, which then can be used through API.

You can try these examples using our playground : https://datadance.app/design/build

Summary

Below is a simple flowchart explaining how datadance works:

graph TD
  Start[Input JSON] --> Init[Copy input to<br> Input & Derived states]
  Init --> Trasnsforms[Trasnsforms]
  Trasnsforms --> |each transform|Transform[Transform]
  Transform --> Evaluate[Evaluate]
  Evaluate --> Update[Update derived state]
  Update --> |Next transform|Trasnsforms
  Trasnsforms --> Condition{All transforms evaluated?}
  Condition -- Yes --> Return[Return Transformed JSON]