.png){kind=icon}
Return
Accelerating Python Code with Parallel I/O
.png){kind=icon}
PRE
AI
DOI：10.1007/978-3-032-02088-8_28.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Python is the dominating data science language, leaving behind other languages like C++, Java and R. Python libraries wrap highly tuned, efficient, accurate C++ and C code for linear algebra, numerical methods and data manipulation. Moreover, the Python runtime works flawlessly across diverse operating systems (Linux, Windows) and CPU architectures, including x86 and ARM. From an accelerator perspective, Python code is processed on multi-core CPUs and GPUs, whose power is not fully exploited. In this paper, we study how to improve Python I/O bottlenecks. We focus on data set summarization to compute a model on a large data set, stored on a CSV file (the most common format used in practice). Heeding these challenges, we introduce two simple, but fundamental, I/O optimizations: parallel multi-threaded read and chunk-based scan (similar to reading file blocks in a DBMS). An experimental validation on different cloud servers, provides a realistic scenario. We show our optimized Python code can work faster than existing Python functions, it exhibits almost linear speed up I/O as more threads are used (up to a limit), but it can still leverage parallel processing for the CPU-intensive floating point computations. To round up our study, we justify chunk size is a critical performance parameter that depends on data set size as well as cloud server configuration.
Keywords:
Python
I/O optimization
parallel processing
data science
chunk-based scan
Journal
D
IF:
0
Papers:
27
Citations:
0
Organization
.png){kind=icon}
