Need help on building a Data Streaming Architecture

Hi, Greetings! ✅checked your project details: building a Data Streaming Architecture ✅Completed Time: In project deadline We have worked on 600 + Projects. I have 6 + years of the experience in same kind of projects. If you are looking for a true Freelancer, I am the Right person for you. I am available almost 24-7 and am very responsive. I feel proud that I am a trusted Freelancer who pleases almost every single client. You can rest assure, your work will be delivered well in advance of others, with passion and accuracy. I guarantee you instant communication & responses when you need me. Why choose me? I think every client is the reason for my success. I only take projects which I am sure I can do quickly. My Portfolio Items: https://www.freelancer.com/u/schoudhary1553 I would really like to work with you on this project. If interested, Kindly contact me via chat for further details and discussion.. Thank you Sandeep Digital screencast

Hi, I have checked your requirements, I can build architecture you need. I will fetch data from API and then store in DB, we can use elastic search or mongodb. You mentioned that you get logs from API. Is it from your log server? I need more details, can we chat?

I have gone thrgh the requirement . its intereting and challenging . We are a Team of members with various programming skills in languages like PHP, .NET , c#, Python,MySQL, SqlServer,posgresql,Cassandra, Bigdata , Artificial Intelligence Etc . I have 15 years of experience in the indusrty . Please PM for further details .

Hello, Having thought about it there's a few ways to go but my concern is that having to use an entire DB, as some sort of intermediary marshaller, is a bit wasteful. Regardless, the options: 1) As you say make use of an upstream API to get the records, sort and filter and write to individual tables in the database. That way each server is locked to and identified by a corresponding table name. 2) Do some form of ETL on the fly. For example if we know to expect a mix of server names in the payload we can do one single query to get the payload and put it in cache (memory, redis, disk, somewhere) and then have a corresponding API that when queried by your program will expose the necessary parameters and abilities for you to filter on that data be it server name, date range and all the other usual things. This removes the need to manage a database and as a non-persistent source gets overwritten or expired via TTL in the case of redis. 3) Possibly fix the upstream API. It is obviously getting the data from somewhere (central log server maybe?) so there must be a possibility to, at that point, inject the required intelligence to enhance its functionality. Well, that's my top three but I would love to explore the options with you in greater detail. Warmest Regards, Stephen Lombard

Hello man i checked all your requirement in details. i have 10+ years in development and I can understand how I can do this with our techniques please come in chat and lets start the work. Qaiser