.. note::
    :class: sphx-glr-download-link-note

    Click :ref:`here <sphx_glr_download_auto_examples_parallel_distributed_backend_simple.py>` to download the full example code
.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_parallel_distributed_backend_simple.py:


Using dask distributed for single-machine parallel computing
=============================================================

This example shows the simplest usage of the dask `distributed
<https://distributed.readthedocs.io>`__ backend, on the local computer.

This is useful for prototyping a solution, to later be run on a truly
distributed cluster, as the only change to be made is the address of the
scheduler.

Another realistic usage scenario: combining dask code with joblib code,
for instance using dask for preprocessing data, and scikit-learn for
machine learning. In such a setting, it may be interesting to use
distributed as a backend scheduler for both dask and joblib, to
orchestrate well the computation.



Setup the distributed client
##############################################################################



.. code-block:: python

    from dask.distributed import Client

    # If you have a remote cluster running Dask
    # client = Client('tcp://scheduler-address:8786')

    # If you want Dask to set itself up on your personal computer
    client = Client(processes=False)







Run parallel computation using dask.distributed
##############################################################################



.. code-block:: python


    import time
    import joblib


    def long_running_function(i):
        time.sleep(.1)
        return i








The verbose messages below show that the backend is indeed the
dask.distributed one



.. code-block:: python

    with joblib.parallel_backend('dask'):
        joblib.Parallel(verbose=100)(
            joblib.delayed(long_running_function)(i)
            for i in range(10))





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    [Parallel(n_jobs=-1)]: Using backend DaskDistributedBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   1 tasks      | elapsed:    0.1s
    [Parallel(n_jobs=-1)]: Done   2 out of  10 | elapsed:    0.1s remaining:    0.5s
    [Parallel(n_jobs=-1)]: Done   3 out of  10 | elapsed:    0.1s remaining:    0.3s
    [Parallel(n_jobs=-1)]: Done   4 out of  10 | elapsed:    0.1s remaining:    0.2s
    [Parallel(n_jobs=-1)]: Done   5 out of  10 | elapsed:    0.1s remaining:    0.1s
    [Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.1s
    [Parallel(n_jobs=-1)]: Done   7 out of  10 | elapsed:    0.1s remaining:    0.1s
    [Parallel(n_jobs=-1)]: Done   8 out of  10 | elapsed:    0.1s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.2s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.3s finished


Progress in computation can be followed on the distributed web
interface, see http://dask.pydata.org/en/latest/diagnostics-distributed.html


**Total running time of the script:** ( 0 minutes  0.732 seconds)


.. _sphx_glr_download_auto_examples_parallel_distributed_backend_simple.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download

     :download:`Download Python source code: distributed_backend_simple.py <distributed_backend_simple.py>`



  .. container:: sphx-glr-download

     :download:`Download Jupyter notebook: distributed_backend_simple.ipynb <distributed_backend_simple.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.readthedocs.io>`_
