python script to automize data file upload to existing table appending the data

Resumen del concurso

Python developer to automize extracting file from ftp loading into Big Query
Basically looking for an ETL pipeline where you automize the complete ETL on a weekly basis

Extract : Data from FTP server (get data file)(csv,Excel....)

Transform : append Data to existing table in BigQuery. Make sure no duplicates, clean data,do not overwite

Load : Load it to Big Query Table

you can create a small summy sample csv file to use.

Habilidades recomendadas

Principales propuestas de este concurso

Ver más participaciones

Tablero de aclaración pública

  • ashikmohann
    ashikmohann
    • 2 años atrás

    hi, the script should be scheduled to run weekly?

    • 2 años atrás
    1. viral15
      Organizador del concurso
      • 2 años atrás

      Yes

      • 2 años atrás
  • AVKor
    AVKor
    • 2 años atrás

    Hi, #extended please and upload a samples of your files (what you are going to get from your FTP server and your database schema).

    • 2 años atrás
    1. AVKor
      AVKor
      • 2 años atrás

      In this case it will be useless to you since the structure, data types, database schema and so on will be not the same as you have and could not be used for your dataset. But if you are going to rewrite the script yourself for your own needs that's OK.

      • 2 años atrás
    2. viral15
      Organizador del concurso
      • 2 años atrás

      Yes I will edit all the schemas and additional details
      We are looking to see who can provide python script that would extract - load the files to BigQuery
      You can use a very simple csv file with few columns...
      Just looking to see how the logic is created to automize the process

      Thank you

      • 2 años atrás
  • viral15
    Organizador del concurso
    • 2 años atrás

    Please not I am specifically looking for a python script

    • 2 años atrás
  • samiul92
    samiul92
    • 2 años atrás

    Hi, I am creating a program for you. So I need to know that from which source of FTP(Any Database like Oracle Database/MS SQL/MySQL or Just like Excel Sheet) to extract data and Load into which Database?

    • 2 años atrás
    1. viral15
      Organizador del concurso
      • 2 años atrás

      Load into Big Query SQL

      • 2 años atrás
    2. samiul92
      samiul92
      • 2 años atrás

      Ok, Thanks for answer

      • 2 años atrás

Mostrar más comentarios

Cómo comenzar con los concursos

  • Publica tu concurso

    Publica tu concurso Fácil y rápido

  • Recibe montones de propuestas

    Consigue toneladas de propuestas De todo el mundo

  • Elige la mejor propuesta

    Elige la mejor propuesta ¡Descarga fácilmente los archivos!

Publica un concurso ahora o únete a nosotros hoy