Did you know that you can interact live with Spark in ADF’s Mapping Data Flow right in the Expression Builder?

First, turn on the debug mode in the ADF browser UI:

Each transformation in your data flow has a Data Preview tab in the bottom Settings pane. You can view the current state of your data as it transforms in this view pane.

Also, within the Expression Builder, we have a view pane with sample results of your expression in the bottom of the window. In fact, you don’t need to use data from your incoming data stream to test expression code.

In these 2 examples, I’m just entering a static value in quotes for a literal string. I’m then testing 2 very tricky expression functions, one is using regular expressions and the other is a date time conversion. While these operations by themselves are rather simplistic, getting them correct and accurate can take a lot of trial & error. This is where the power of testing expressions in the ADF Mapping Data Flow expression builder is very powerful.