Feature #5295

Implementation of multi-threaded rsync for base backups

Added by Joshua A. Drake 04/17/2014 (almost 8 years) ago. Updated 05/15/2014 (over 7 years) ago.

Status:ResolvedStart date:04/17/2014
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Resolution:

Description

Due to the single-threaded nature of pg_basebackup, performing a base backup can be a rather slow process.
Multi-threading the interface in Python would resolve this.

threaded_rsync.py Magnifier (7.45 KB) Joshua Drake, 04/17/2014 09:11 AM

History

#1 Updated by Joshua Drake 04/17/2014 (almost 8 years) ago

On 04/17/2014 09:08 AM, wrote:

----------------------------------------
Due to the single-threaded nature of pg_basebackup, performing a base backup can be a rather slow process.
Multi-threading the interface in Python would resolve this.

Here is the code to implement.

--
Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc
Political Correctness is for cowards.

threaded_rsync.py

#2 Updated by Joshua A. Drake 04/17/2014 (almost 8 years) ago

Development plan:

  • Add value rsync_threads (int) to cmd_standby.ini; Default 1
    - Corresponds to --num_threads options passed to threaded_rsync.py. Defines amount of threads used in parallel rsync for base backups.
  • Update doc/cmd_standby.README accordingly
  • In cmd_standby: CMDStandby.base_backup_func, insert threaded_rsync.py with appropriate options

#4 Updated by Joshua A. Drake 04/17/2014 (almost 8 years) ago

  • Status changed from New to Resolved

#5 Updated by Joshua A. Drake 05/07/2014 (over 7 years) ago

threaded_rsync.py is not always located correctly as it is referred to by a relative path
Fix: https://github.com/commandprompt/PITRTools/pull/41

#6 Updated by Joshua A. Drake 05/15/2014 (over 7 years) ago

threaded_rsync.py is not invoked "properly" in subprocess.call(), and thus only works sometimes.
Fix: https://github.com/commandprompt/PITRTools/pull/43

Also available in: Atom PDF