Comparing file tables with DiffKit

In order for DiffKit to diff two Tables, a Table simply being a set of rows, it must know how to align the rows from the left side Table with those on the right. It does that using a key; one or more columns. Before the row sets can be diff’d, they must be sorted. If you are using a DB Source, DiffKit will sort the Tables for you and you don’t need to do anything. If you are using a File Source, DiffKit will not sort the files for you. In the future, I plan to modify DiffKit to do the sorting for you, but in the meantime you must sort the files yourself.

When you perform the sort, you need to ensure that you are sorting using the same comparison function that DiffKit will use internally to compare rows. That’s because comparing rows internally is how DiffKit figures out ROW_DIFFs.

If you are using a MagicPlan to diff File Sources, DiffKit has no data type information about the columns; MagicPlan doesn’t allow it. It’s just a text file, so DiffKit has to assume that all columns are data type String. In that case, DiffKit will use a lexical (String) sort internally to compare rows. And you must ensure that you have also used a lexical sort when you sort the file. I believe that the default comparison term for Unix sort is lexical.

If you are using a PassthroughPlan to diff File sources, you need to tell DiffKit what are the types of each column. If in the PassthroughPlan you have told DiffKit that the key column is type String, then DiffKit behaves exactly as in the case of the MagicPlan, and you must use a lexical sort on the file. However, if in the PassthroughPlan you tell DiffKit that the key has a numeric data type, then DiffKit will internally use a numeric comparison on the rows, and you must sort the file using a numeric comparison.

Bottom line:

DiffKit internal comparison ==(must equal)== comparison used to sort File. DiffKit internal comparison is based on data type(s) of the key.

MagicPlan always results in String data type(s) for the key. PassthroughPlan results in whatever column data types you specify for the key.