plhw / tags / science & technology

Tagged with “science & technology” (5)

  1. Deep Dive into GPU Support in Apache Spark 3.x

    Utilizing accelerators in Apache Spark presents opportunities for significant speedup of ETL, ML and DL applications. In this deep dive, we give an overview of accelerator aware task scheduling, columnar data processing support, fractional scheduling, and stage level resource scheduling and configuration. Furthermore, we dive into the Apache Spark 3.x RAPIDS plugin, which enables applications to take advantage of GPU acceleration with no code change. An explanation of how the Catalyst optimizer physical plan is modified for GPU aware scheduling is reviewed. The talk touches upon how the plugin can take advantage of the RAPIDS specific libraries, cudf, cuio and rmm to run tasks on the GPU. Optimizations were also made to the shuffle plugin to take advantage of the GPU using UCX, a unified communication framework that addresses GPU memory intra and inter node. A roadmap for further optimizations taking advantage of RDMA and GPU Direct Storage are mentioned. Industry standard benchmarks and runs on production datasets will be shared.

    About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business. Read more here: https://databricks.com/product/unifie

    Connect with us: Website:…

    ===
    Original video: https://www.youtube.com/watch?v=4MI_LYah900&feature=emb_title
    Downloaded by http://huffduff-video.snarfed.org/ on Sun Aug 2 19:32:43 2020 Available for 30 days after download

    —Huffduffed by plhw

  2. Secure, private environments in the cloud & on-prem with Virtual Clouds (Google Cloud Next ‘17)

    Google Cloud allows you to expand with on-demand capabilities as you need them, but how can your organization enjoy those advanced capabilities in a secure, private environment that’s completely isolated, policy controlled and automated and allows granular delegation to your developers and administrators? In this video, Ines Envid, Neha Pattan, and Stefan Lindblom walk you through how Google Cloud Virtual Private Cloud (VPC) provides you with a secure and flexible sandbox to run your cloud workloads and federate with on-premises applications. You’ll learn how to use Cloud VPC to enable private integration with the Google Cloud Machine Learning, storage, and Big Data services.

    Missed the conference? Watch all the talks here: https://goo.gl/c1Vs3h Watch more talks about Infrastructure & Operations here: https://goo.gl/k2LOYG

    ===
    Original video: https://youtu.be/o_DJ4L4lyKE?t=36m22s
    Downloaded by http://huffduff-video.snarfed.org/ on Thu, 12 Apr 2018 17:08:02 GMT Available for 30 days after download

    —Huffduffed by plhw

  3. N Languages in N Months: Functional Fall–Elm - John Mayer

    For more tech talks and to network with other engineers, check out our site https://www.hakkalabs.co/logs

    Full post here:

    This talk by John Mayer is an entirely-live-coded workshop on Elm and its core features: typed functional programming, declarative graphics, and functional reactive programming (FRP) with Signals. John also introduces a toolkit for creating interactive user interfaces for the browser.

    ===
    Original video: https://www.youtube.com/watch?v=e_pw9YMzCMM
    Downloaded by http://huffduff-video.snarfed.org/

    —Huffduffed by plhw

  4. Functional reactive programming in Elm

    FRP can be defined as a λ-calculus that admits temporal types, i.e. types given by a propositional intuitionistic linear-time temporal logic (LTL). Although the Elm language uses only a subset of LTL, it achieves high expressivity for GUI programming. I will formally define the operational semantics of Elm. I discuss the current limitations of Elm and outline possible extensions. I also review the connections between temporal logic, FRP, and Elm. Talk slides: https://github.com/winitzki/talks/blob/master/elm-talk/elm_talk.pdf

    ===
    Original video: https://www.youtube.com/watch?v=DiZ1CfLQvIU
    Downloaded by http://huffduff-video.snarfed.org/

    —Huffduffed by plhw