Loading incremental data
On this section, you'll learn how to create and execute an incremental manual loading job to the Data Loading & Preparation module.
Create a data source!
Before you create a new Loading Job, you'll need to have at least one Data Source configured. If you didn't create a Data Source yet, read the Connecting data sources guide to create one.
Creating an incremental manual loading job
The manual incremental loading job loads your data on-demand when new data needs to be inserted in SlicingDice. You need to manually start this job. Differently from one-time loading job, only new rows will be inserted in the database.
Column with timestamps!
For this guide, your dataset must have a timestamp column (Date, Datetime), or the Manual Incremental Loading Job won't work. This column serves to indicate SlicingDice what data is new and should be inserted to your database.
The first step is to configure your loading job identification as you can see below.
Here you'll define your loading job name, description, the registered data source to connect and the job type (Manual Incremental in this case). When ready, click on Save & Continue.
Now you'll define in which of your databases and dimensions the data will be loaded.
When everything is set, click on Save & Continue.
After this you'll map your source columns and match them with the SlicingDice columns and their supported data types.
For it, the first thing you should do is to detect your source schema, selecting the data you want to retrieve. You can do it choosing the Access mode - query or table - where you can define a SQL query or just inform the name of the table to retrieve data.
After detecting your schema, the selected source columns will appear at the Attribute and Event Columns mapping section. Map your source's columns to SlicingDice's existing columns or create new ones if needed. Also define the column to use on incremental job that is necessary to guarantee that just new data will be loaded. After that click on Save & Continue.
Finally just check your loading job configuration and click on Submit. Now you'll be able to find your created loading job at the loading jobs list. That's it! The next step is to execute your loading job manually.
Executing and checking an incremental manual loading job status
Once the loading job is created, it will automatically appear on the list of loading jobs. As your Loading Job is set as Manual Incremental Job just start it by on the Play button at the Actions section on your loading jobs list.
You can also see the Status of your Loading Job changing to Running.
Time to execute the job!
The Loading Job execution time can vary by the size of your dataset and the chosen Rows per request per second configured at step 3. When the "Status" parameter modifies to "Completed", you can start querying your data.