Afternoon,
I would like to create a data pipeline using Azure Data Factory between json files which appear in my Azure Blob containers and my Azure Sql Tables.
My blob currently has 3 containers and receives dates from a web app from API pulls from three platforms, Square, Deputy and Xero. The data in each container is subtly different but all in json.
Part One:
I would like you to create a data pipeline for the data in the square container which appends to an Sql Table you create. There should be 3 pipelines and 3 tables.
1. Historical Data and Live Data is placed into the same sql table and simply appended.
2. Modifier List is placed into it’s own sql table. A new modifier list file will appear every week and it should replace the data in the it’s sql table. It will always have the same schema & columns, it might just have a different number of rows and values.
3. Item List is placed into it’s own sql table. A new item list file will appear every week and it should replace the data in the it’s sql table. It will always have the same schema & columns, it might just have a different number of rows and values.
Question: Are you able to achieve this with the data as they are represented in the containers at the moment or would it be easier for each pipeline to have it’s own container store? The Historical data files name will always begin with “Historical_Data_”. The modifier list will always start will always contain “_ModifierLists_” in the name. The Item List will always contain “_Items_” in the name.
Part Two:
I would like you to create a data pipeline for the data in the xero container. The data will flow into the container every week, it will contain the same schema but maybe different values and these should just be appended to the sql table you create.
Part Three:
I would like you to create a data pipeline for the data in the deputy container. The data will flow into the container every few days, it will contain the same schema but maybe different values and these should just be appended to the sql table you create.
I have already attempted to create the pipeline myself but I’m unable to expand the nested arrays in a manner as they work completely. The Xero Component (Part Two) will most likely be the most time intensive as there are quite a few nested arrays, but all of which follow the same format.
I’ve included my sql query code for creating a table for the Square Historical and Live Data if you’d like to use that to save you some time in writing the sql code.
Part One Files (Square):
Historical Data: Name: Historical_Data_Neoscona_2017-02-18 23:27:39txt Download Link: [login to view URL]!At8Q-ZbRnAj8hkBXyPhT6jyZ-2vz
Live Data: Name: shilpa9_99@_4KVK34WXWP2G9_G66zuktspUh1ha8eaFubLQB_2017-02-18 23:31:23txt Download Link: [login to view URL]!At8Q-ZbRnAj8hj8TStM0PrDgaajR
Sql Query Code Download Link: [login to view URL]!At8Q-ZbRnAj8hj32ErBpxz31EIHI
Part Two (Xero) Files:
Data File Name Historical_Data_Demo_-_Company_-_(Global)_2017-02-18 23-33-18txt (1) Download Link: [login to view URL]!At8Q-ZbRnAj8hkEAaL0YhtMfnl4C
Part Three Files (Deputy):
Data File Name: Historical_Data_Test_2017-02-18 23-35-19txt Download Link: [login to view URL]!At8Q-ZbRnAj8hj7JBMpmJLe35EgS
Hi,
Answer: It doesn't matter if files are in same container or is separate container as long as file name can be identified using run time values (e.g. date, time etc.).
I think the challenge with your project is to deal with dynamic number of "JournalLines" in historical in Xero.
I have 4+ years of experience on Azure (total 11+) and I have been recently started working on DF.
Please ping me to discuss more on your project.
Thanks
Prasoon
Hello,
Im interested to work on this. i have gone through the high level job description.
If you are interested, please let me know.
We will discuss and start the project.
Thank you.
Regards,
Sheik
Hi, I will work as per your direction. I ensure you the accuracy of my work. I am ready to start working right now.I will show you sample to make you sure about the quality of my work. I am waiting for your response to start working. Thanks