Loading data incrementally
On this section, you'll learn how to create and execute an incremental manual loading job to the Data Loading & Transformation module.
Create a data source!
Before you create a new Loading Job, you'll need to have at least one Data Source configured. If you didn't create a Data Source yet, read the Connecting data sources guide to create one.
Creating an incremental manual loading job
The manual incremental loading job loads your data on-demand when new data needs to be inserted in SlicingDice. You need to manually start this job. Differently from one-time loading job, only new rows will be inserted in the database.
Columns for incremental insertions!
On incremental jobs, SlicingDice needs a way to identify which data is new and which isn't. You must define a column for this when creating an Incremental Loading Job. This column's type can be numeric (e.g. integer, int, smallint, bigint, decimal, real, numeric or float), date or datetime column.
This way SlicingDice tracks what data has already been loaded and it should load now, whenever you run an incremental loading job.
In the Control Panel, to go to the Loading Jobs page and click on Create New Loading Job to start creating your job.
The first step is to configure your loading job identification as you can see below.
Here you'll define your loading job name, description, the registered data source to connect and the job type (Manual Incremental in this case). When ready, click on Save & Continue.
Now you'll define in which of your databases and dimensions the data will be loaded.
When everything is set, click on Save & Continue.
After this you'll map your source columns and match them with the SlicingDice columns and their supported data types.
For it, the first thing you should do is to detect your source schema, selecting the data you want to retrieve. You can do it choosing the Access mode - query or table - where you can define a SQL query or just inform the name of the table to retrieve data.
After detecting your schema, the selected source columns will appear at the Attribute and Event Columns mapping section. Map your source's columns to SlicingDice's existing columns or create new ones if needed. Also define the column to use on incremental job that is necessary to guarantee that just new data will be loaded. After that click on Save & Continue.
Mapping an Entity ID is mandatory!
On SlicingDice an Entity ID column behaves similarly to a primary key column in a database with unique values stored on it, so any data you insert will always be associated with a single identifier. This is why you MUST define one of your columns from your sources to be your Entity ID column, otherwise you'll receive an error while creating your loading job.
Finally just check your loading job configuration and click on Submit. Now you'll be able to find your created loading job at the loading jobs list. That's it! The next step is to execute your loading job manually.
Executing and checking an incremental manual loading job status
Once the loading job is created, it will automatically appear on the list of loading jobs. As your Loading Job is set as Manual Incremental Job just start it by pressing on the Play button at the Actions section on your loading jobs list.
You can also see the Status of your Loading Job changing to Running.
Time to execute the job!
The Loading Job execution time can vary by the size of your dataset and the chosen Rows per request per second configured at step 3. When the "Status" parameter modifies to "Completed", you can start querying your data.