WEKO3
アイテム
A High-performance Deadlock-free Overlay for Wide-area Parallel and Distributed Programming
http://hdl.handle.net/2261/28050
http://hdl.handle.net/2261/280504310d275-9979-4e95-a2e9-2a21115ead2e
| 名前 / ファイル | ライセンス | アクション |
|---|---|---|
hironaka.pdf (1.4 MB)
|
|
| Item type | 学位論文 / Thesis or Dissertation(1) | |||||
|---|---|---|---|---|---|---|
| 公開日 | 2011-08-08 | |||||
| タイトル | ||||||
| タイトル | A High-performance Deadlock-free Overlay for Wide-area Parallel and Distributed Programming | |||||
| 言語 | ||||||
| 言語 | eng | |||||
| 資源タイプ | ||||||
| 資源 | http://purl.org/coar/resource_type/c_46ec | |||||
| タイプ | thesis | |||||
| その他のタイトル | ||||||
| その他のタイトル | 広域並列分散プログラミングのための高性能デッドロックフリーオーバレイ | |||||
| 著者 |
Hironaka, Ken
× Hironaka, Ken |
|||||
| 著者別名 | ||||||
| 識別子Scheme | WEKO | |||||
| 識別子 | 5667 | |||||
| 姓名 | 弘中, 健 | |||||
| 著者所属 | ||||||
| 著者所属 | 情報理工学系研究科 電子情報学専攻 | |||||
| Abstract | ||||||
| 内容記述タイプ | Abstract | |||||
| 内容記述 | Parallel and distributed computing in Wide-area environments is complicated by connectivity issues like NATs/firewalls, and dynamic changes in the available resources. Therefore, programming in these environments requires substantial support from programming frameworks. Such frameworks can help resolve low-level concerns while providing abstractions to implement applications more easily. Meanwhile overlay networks, which create their own network on top of the internet using UDP/TCP links, have become popular communication mediums for frameworks in these environments as they transparently enable communication even in wide-area networks. Thus, high-performance overlays are crucial in this context. This paper presents one such programming framework and a high-performance overlay whose goal is to serve as its communication layer./ The first proposal designs and implements an overlay-based programming framework for largescale wide-area computing by minimally extending distributed object-oriented models for maximum generality and flexibility. Aside from introducing constructs to facilitate parallel computation and to handle runtime process joins/leaves, the framework automatically creates and manages an overlay to enable communication across WANs with minimal user intervention. In the framework, parallelism is expressed via asynchronous method invocations to distributed objects for a natural transition from sequential programs. To make parallelism manageable, the framework introduces an implicit serialization semantics on objects to relieve programmers from explicit synchronization, while avoiding deadlock problems. In the implementation, participating nodes automatically construct an overlay, and relieve the users from manual configuration and managing the overlay as it dynamically changes during the computation. This framework, gluepy is implemented as a library for Python to allow rapid development of complex workflows and to maximally leverage the richness of its libraries. For evaluation, this work shows, on over 900 cores across 9 clusters with complex networks (involving NATs and firewalls), how applications with dynamic node joins/failures can be expressed simply and executed easily./ The second proposal presents the design and implementation of a high-performance overlay network that attains the throughput performance of the underlying wide-area network. The addressed core issue is a flow control problem where intermediate overlay nodes have limited buffer memory, while the forwarding must yield full network throughput exceeding Gbps. Implementing a naive flow control, however can deadlock the overlay. The proposed overlay presents a deadlock-free overlay that couples TCP connections and fixed intermediate buffer memory. The method fully takes advantage of TCP’s flow control and implements a simple flow control scheme by creating dependencies among connections in an overlay path. Meanwhile, the proposal adapts a deadlock-free routing algorithm forheterogeneous wide-area networks, so deadlocks among communication can be avoided without sacrificing performance. The proposal also incorporates overlay construction and routing optimizations that account for underlying network latency and bandwidth information. This work demonstrates, via simulation on 13 clusters (515 nodes) and by evaluation on 7 clusters (170 nodes), that the proposed deadlock-free routing poses negligible overhead in comparison to deadlock-unaware routing, and comparably with direct communication. It further demonstrates that for certain collective communications, the proposed overlay even out-performs direct communication by mitigating or completely avoiding network contention. This is shown on systems ranging from a single-switch cluster with 36 nodes to a Grid environment with 4 clusters and 291 nodes. | |||||
| 書誌情報 | 発行日 2009-03-31 | |||||
| 日本十進分類法 | ||||||
| 主題Scheme | NDC | |||||
| 主題 | 548 | |||||
| 学位名 | ||||||
| 学位名 | 修士(工学) | |||||
| 学位 | ||||||
| 値 | master | |||||
| 研究科・専攻 | ||||||
| 情報理工学系研究科電子情報学専攻 | ||||||
| 学位授与年月日 | ||||||
| 学位授与年月日 | 2009-03-31 | |||||
