Project

General

Profile

Feature #5306

Reimplementation of queue

Added by Joshua A. Drake over 10 years ago. Updated about 10 years ago.

Status:
Feedback
Priority:
Normal
Assignee:
Joshua Drake
Start date:
05/13/2014
Due date:
% Done:

0%

Estimated time:
Resolution:

Description

Reimplementation of PITRTools queue
The current queue system possesses the following limitations:
1. No job control - If we are generating xlogs faster than we can ship them, rsync processes will be spawned uncontrollably.
2. A continuous stream of transaction logs must be generated for correct operation of the archiver. If there are no new logs
coming in, logs still in the queues will not get shipped.

Suggested solution:

Queue management daemon - Have cmd_archiver only queue logs, with a separate daemon shipping them.
  * cmd_queue daemon 
    - write PID to a file
    - In worker thread
      ] Ship logs in queue with subprocess calls when self to-do list is not empty
        } If successful, remove paths from self to-do list and log appropriate output
        } Else, log which queues failed 
      ] Wait a few seconds [configurable]
      ] Repeat
    - In main thread 
      ] Wait for data [absolute paths to queued logs] on a pipe or UNIX socket
      ] On data, parse and extract paths, append paths to self to-do list
      ] Repeat
  * When cmd_archiver is called with a new archive, copy the archive to slave queues and write paths to cmd_queue via pipe/socket

Also available in: Atom PDF