Mastering a data pipeline with Python / Robson Luis Monteiro Junior (Microsoft)

Приглашаем на Moscow Python Conf 2023, которая пройдет 19 и 20 мая 2023 в Москве в рамках Positive Hack Days. Программа, подробности и билеты по ссылке -------- Python Conf 2020 Online Тезисы и презентация: Building data pipelines are a consolidated task, there are a vast number of tools that automate and help developers to create data pipelines with few clicks on the cloud. It might solve non-complex or well-defined standard problems. This presentation is a demystification of years of experience and painful mistakes using Python as a core to create reliable data pipelines and manage insanely amount of valuable data. Let’s cover how each piece fits into this puzzle: data acquisition, ingestion, transformation, storage, workflow management and serving. Also, we’ll walk through best practices and possible issues. We’ll cover PySpark vs Dask and Pandas, Airflow, and Apache Arrow as a new approach. ------— Нашли ошибку в видео? Пишите нам на support@