Stories Tag: dask

Prefect x Kubernetes x Ephemeral Dask: Power, without responsibility | by George Coyne | SlateCo | Jul, 2021 | Medium

admin 1 week ago in

Prefect is rapidly becoming the standard in dataflow automation. In our last blog we briefly ran through using Prefect Executors to parallelize on a single node. The question that usually follows single node parallelization is: How do we parallelize across multiple nodes. The goal of this blog is to provide a minimal example of how to(with as little external configuration as possible) parallelize across multiple…

Faster ML platform with Kubernetes, Kubeflow, Dask, Rapids, Bokeh… no vendor lock-in | by Michael Sebbah | Oct, 2020 |...

admin 9 months ago in

Data scientists expect modern IT tools to compute ml pipelines, handle large datasets, and share data finely. Python tools(Pandas, Numpy, Pytorch, Keras, Jupyter …) are common in this area, thus users are keen on playing with a pythonic platform. Several vendors propose a solution, but let me explain why open source services are better off. For modern high performance computing, workflow processing, APIs, webapps, in…

Managing Pangeo Environments for Distributed Computing

admin 1 year ago in

By Tom Augspurger, Scott Henderson, and Joe Hamman Managing computational environments has been a persistent challenge for both administrators and users of Pangeo distributed computing infrastructure. While there are still improvements to be made, we’re reasonably happy with our current setup. This post will discuss But first: what do we mean by environment management? Roughly speaking, we’re talking about the set of packages (typically Python…