Data Engineering Specialist Needed
$250-750 USD
Paid on delivery
I'm looking for a data engineer with solid Pyspark knowledge to assist in developing a robust data storage and retrieval system, primarily focusing on a Data Warehouse.
Key Responsibilities:
- Implementing efficient data storage solutions for long-term retention and retrieval
- Ensuring data quality and validation procedures are in place
- Advising on real-time data processing capabilities
Ideal Candidate:
- Proficient in Pyspark with hands-on experience in data storage and retrieval projects
- Familiar with Data Warehousing concepts and best practices
- Able to recommend and implement appropriate real-time processing solutions
- Strong attention to detail and commitment to data quality.
Specifically, I have a Jira ticket that consists of creating an application that runs on Airflow and connects to an API with survey metadata with values such as if it was opened, if it was answered, etc, then it should generate a zip file with all the data in jsons and save it in a S3 bucket. Once in the bucket you must save the information in a Hive table.
All the code should be from Pyspark and there is a similar application that saves the raw survey data that you can take as reference.
You need to create the table and generate the code, the end client is Expedia and it should be done using their environment, I would give you the credentials and ask what you need with my colleagues.
If you like we can have a video call to explain everything better.
Project ID: #38088526
About the project
13 freelancers are bidding on average $463 for this job
With my background as an experienced developer and a passion for data engineering, I believe I'm the ideal candidate for this job. My proficiency in Pyspark and data storage and retrieval projects is evident from my ac More
Hi Cecilio C., How are you doing? As a professional mechanical and civil expert with expertise inPySpark, GitHub and Hive, I eagerly anticipate the opportunity to complete this project for you. Please drop me a message More
Hi there,I'm biddin on your project "Data Engineering Specialist Needed"GitHub, Hive and PySpark I'm looking for a data engineer with solid Pyspark knowledge to assist in developing a robust data storage and retrieval More
As a Senior Full Stack Engineer and Team Lead, my primary objective has always been to provide superior service while maintaining cost-efficiency. With 12 years of successful projects under my belt, including numerous More
As a Senior Full Stack Engineer with over 12 years of experience, I have inevitably grown to become highly proficient in a multitude of tools and technologies that are relevant to your project, including PySpark and Gi More
Hi I am a Data Engineer having strong experience in PySpark. Hope to discuss the project details. Thank you
Hello, I have carefully read your project requirement. With my proficient in Pyspark, I am sure, I can fulfill your requirement and get desired results. I am ready to start the project right away. Regards, Bhupendra
I am confident in my ability to develop a robust data storage and retrieval system using Pyspark for a Data Warehouse project. My solution includes implementing data quality checks, real-time processing, seamless integ More
Hi, I have +7 years of experience dealing with machine learning algorithms and worked on multiple projects in this field, Please contact me to discuss more. Have a nice day
Hi there, I am a data engineer having 8 years of experience in data engineering, data warehousing, ETL using technologies like Python, Pyspark, SQL, AWS services, RDBMS etc. I have worked on different kind of problems/ More
