"Taming the Dragon": Data Flow Programming with Zenoh-Flow
•
0 likes•17 views
Welcome to the sixth and last webinar in the "Taming the Dragon" series about Zenoh and its use for robotics, autonomous vehicle and Internet-scale HPC communities. In this webinar, Julien Loudet (our Product Conductor) and Gabriele Baldoni (our Runtime Lead) will talk about Zenoh-Flow and also show a hands-on demonstration of its capabilities. You can read more about Zenoh-Flow here: https://github.com/eclipse-zenoh/zenoh-flow More details about configuring and starting a Zenoh-Flow: https://github.com/eclipse-zenoh/zenoh-flow/wiki/ Stay up to date with the latest news: Twitter: https://twitter.com/zettascaletech LinkedIn: https://www.linkedin.com/company/zettascaletech/ Website: https://www.zettascale.tech/ Newsletter: http://eepurl.com/igPw31
Recommended
More Related Content
Similar to "Taming the Dragon": Data Flow Programming with Zenoh-Flow
Similar to "Taming the Dragon": Data Flow Programming with Zenoh-Flow (20)
More from ZettaScaleTechnology
More from ZettaScaleTechnology (8)
Recently uploaded
Recently uploaded (20)
"Taming the Dragon": Data Flow Programming with Zenoh-Flow
- 1. Product Conductor Julien LOUDET, PhD Taming the Dragon - Ep. 6 Zenoh-Flow Runtime Lead Gabriele BALDONI julien.loudet@zettascale.tech gabriele@zettascale.tech
- 2. Zenoh Seamless communications for the Cloud-to-Thing continuum, that elegantly blends pub/sub, queries and geo-distributed storages, that is performant, e ffi cient and extensible routing
- 3. Zenoh- Flow Extensible Enhance zenoh with plugins Interact with other technologies SSE …
- 5. Zenoh-Flow?
- 6. Data fl ow Programming Data fl ow Programming is a programming paradigm that models a program as a directed graph of the data fl owing between operations Data fl ow Programming languages share some features of functional languages, and were generally developed in order to bring some functional concepts to a language more suitable for numeric processing
- 7. Relevance The data fl ow programming paradigm was introduced back in the 60’s at MIT by Jack Dennis Since its introduction it has been used as a natural paradigm to compose applications that operate on streams of data Data fl ow programming is gaining back relevance thanks to the growth of robotics and in general data-processing applications found in autonomous vehicles
- 8. Key principles Data fl ow programming organises application as a directed graph of operators through which data fl ows A B C D E 8
- 9. A B C D E Thing Edge Cloud Data fl ow × Cloud-to-Thing
- 10. A B C D E Data fl ow × Zenoh Thing Edge Cloud
- 12. Zenoh’s data fl ow programming framework for Cloud-to-Thing applications Uni fi ed abstraction, automated & location-transparent deployment of the computing units Built in Rust, has high-performance, optimises (zero-copy) communications, low latency Zenoh-Flow
- 13. Terminology
- 14. Data fl ow In Data fl ow programming, a data fl ow represents your entire application. A data fl ow is comprised of a set of nodes, connected by directed links. A data fl ow receives a set of inputs and produces a set of outputs. In Zenoh-Flow, a data fl ow is deterministic: the same set of inputs will produce the same set of outputs.
- 15. Nodes A node is a computing unit. Zenoh-Flow supports 3 types of node: ‣ Source: they only send data to downstream nodes. Sources are the entry points of your data fl ow. ‣ Sink: they only receive data from upstream nodes. Sinks are the exit points of your data fl ow. ‣ Operator: they perform transformations on the data. This is where the business logic of your application is located. O So Si
- 16. Links A link connects two nodes. Speci fi cally, a link connects an output to an input of the same type*. Not all nodes possess inputs and outputs! ‣ Source: only possesses outputs. ‣ Sink: only possesses inputs. ‣ Operator: possesses both. Si
- 17. Implementation
- 18. Our goal Thing Edge Cloud A E B C D
- 19. Creating nodes Thing Edge Cloud A E B C D
- 20. Creating nodes In Zenoh-Flow, a node is the combination of a descriptor and an implementation (a shared library or a Python script).
- 23. Data fl ow descriptor Thing Edge Cloud A E B C D
- 24. Data fl ow descriptor The descriptor of a data fl ow is a contract that Zenoh-Flow will enforce and that stipulates: ‣ the nodes involved, i.e. where they are de fi ned, ‣ the links, i.e. how the nodes are connected, ‣ the mapping, i.e. where each node will run.
- 26. Deployment
- 27. Deployment Thing Edge Cloud A E B C D
- 28. Deployment ‣ For each host, start a Zenoh router with the Zenoh-Flow plugin properly con fi gured ‣ Make sure that all routers can communicate to each other ‣ Use the CLI zfctl to launch your data fl ow
- 29. Thing Edge Cloud A E B C D
- 30. Thing Edge Cloud A E B C D
- 31. Thing Edge Cloud A E B C D
- 32. Thing Edge Cloud A E B C D
- 33. Thing Edge Cloud A E B C D
- 34. Hands-on
- 35. Converting °F to °C Imagine we have a temperature sensor producing measures in °F and in our system we want to use it in °C. Zenoh-Flow will help us in doing this conversion directly inside a Zenoh router. (X-32)*(5/9) °F °C
- 36. Complex applications: AVs SRC_OPEN_D RIVE open_drive src_lidar_sensor point_cloud src_center_came ra image src_imu Localisation imu_data src_gnss gnss_data ground_pose Sensors Time to decision ttd obstacle_locati on_finder src_gnss_pose _receiver obstacle_locati on_history static_obstacle _location_finde r behavior_plan ning Planner Controller Obstacle detection tf Object tracker Linear predictor Traffic Light detection src_tl_camera src_tl_lidar Waypoints src_global_traject ory point_cloud image Obstacles rt tracked_obstacles tracker_delay Obstacles w location obstacle_trajectories Obstacle predictions Traffic lights Obstacles w location trajectory Waypoints sink driver control_data Perception Planning Localisation Control Prediction
- 38. Complex applications: Montblanc ‣ 2 Sources ‣ 1 Sink ‣ 20 Operators ‣ 36 Links 👩💻 GitHub
- 39. Features
- 40. Performance Nodes on the same daemon will automatically exchange data through channels (without explicit con fi guration!) instead of going through the network and (de)serialising Latency ~2 µs Throughput ~1.6M msg/s Test bed: - AMD Ryzen 5800X CPU, fi xed 4.0 GHz - 32GB of DDR4 3200MHz RAM - 100GbE network card - Ubuntu 20.04.3
- 41. Composite Operators For data fl ow with many nodes, a single YAML is too restrictive and hard to deal with We allow splitting the de fi nition of a data fl ow in multiple fi les
- 42. Next steps ‣ 🪪 Investigate additional type support (e.g. ProtoBuf, Cap’n’Proto) ‣ 🧑💻 Improve developer experience: improve logging, better errors, allow testing of nodes in isolation ‣ 📒 Node registry: o ff er a way to publish and fetch nodes from a “database” ‣ 🏷 Tag-based deployment: remove the need to explicitly specify where each node is running, by, for instance, associating tags to daemons and nodes
- 43. Don’t forget to visit Zenoh’s website… https://zenoh.io/ … Zenoh-Flow’s GitHub … https://github.com/ eclipse-zenoh/zenoh- fl ow … and to join Zenoh’s Discord server! https://discord.gg/ 2GJ958VuHs
- 44. Patience, persistence and perspiration make an unbeatable combination for success. Thank You