Stories Tag: dask

Распределенное МО с Dask и Kubernetes на GCP | by Roman | NOP::Nuances of Programming | Oct, 2021 | Medium

admin 2 weeks ago in

В Sarus мы создаем инструменты для работы с конфиденциальными данными. Наш продукт базируется на 3 “китах”: Таким образом, большую часть времени мы посвящаем тому, что обучаем модели машинного обучения генерировать синтетические данные. Кроме того, мы разрабатываем инструменты для дистанционного выполнения кода. Dask так или иначе связан с обеими вышеуказанными задачами. Это отличная библиотека для параллельных вычислений на Python, а также хорошо разработанное программное обеспечение и…

Executors in Airflow. By Hiren Rupchandani & Mukesh Kumar | by INSAID | Aug, 2021 | Medium

admin 2 months ago in

By Hiren Rupchandani & Mukesh Kumar Airflow executors are the mechanism that handles the running of tasks. Airflow can only have one executor configured at a time; they can be set in the airflow.cfg file in the airflow directory. Primarily, there are two types of executors in Airflow — local executors which run tasks on the same machine (or process) as the scheduler and remote…

Prefect x Kubernetes x Ephemeral Dask: Power, without responsibility | by George Coyne | SlateCo | Jul, 2021 | Medium

admin 3 months ago in

Prefect is rapidly becoming the standard in dataflow automation. In our last blog we briefly ran through using Prefect Executors to parallelize on a single node. The question that usually follows single node parallelization is: How do we parallelize across multiple nodes. The goal of this blog is to provide a minimal example of how to(with as little external configuration as possible) parallelize across multiple…

Faster ML platform with Kubernetes, Kubeflow, Dask, Rapids, Bokeh… no vendor lock-in | by Michael Sebbah | Oct, 2020 |...

admin 12 months ago in

Data scientists expect modern IT tools to compute ml pipelines, handle large datasets, and share data finely. Python tools(Pandas, Numpy, Pytorch, Keras, Jupyter …) are common in this area, thus users are keen on playing with a pythonic platform. Several vendors propose a solution, but let me explain why open source services are better off. For modern high performance computing, workflow processing, APIs, webapps, in…

Managing Pangeo Environments for Distributed Computing

admin 1 year ago in

By Tom Augspurger, Scott Henderson, and Joe Hamman Managing computational environments has been a persistent challenge for both administrators and users of Pangeo distributed computing infrastructure. While there are still improvements to be made, we’re reasonably happy with our current setup. This post will discuss But first: what do we mean by environment management? Roughly speaking, we’re talking about the set of packages (typically Python…