pgsql: Add a Gather executor node.

Add a Gather executor node.

A Gather executor node runs any number of copies of a plan in an equal
number of workers and merges all of the results into a single tuple
stream. It can also run the plan itself, if the workers are
unavailable or haven't started up yet. It is intended to work with
the Partial Seq Scan node which will be added in future commits.

It could also be used to implement parallel query of a different sort
by itself, without help from Partial Seq Scan, if the single_copy mode
is used. In that mode, a worker executes the plan, and the parallel
leader does not, merely collecting the worker's results. So, a Gather
node could be inserted into a plan to split the execution of that plan
across two processes. Nested Gather nodes aren't currently supported,
but we might want to add support for that in the future.

There's nothing in the planner to actually generate Gather nodes yet,
so it's not quite time to break out the champagne. But we're getting
close.

Amit Kapila. Some designs suggestions were provided by me, and I also
reviewed the patch. Single-copy mode, documentation, and other minor
changes also by me.

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/3bd909b220930f21d6e15833a17947be749e7fde

Modified Files
--------------
doc/src/sgml/config.sgml | 46 ++++
src/backend/commands/explain.c | 19 ++
src/backend/executor/Makefile | 4 +-
src/backend/executor/execAmi.c | 8 +
src/backend/executor/execMain.c | 3 +
src/backend/executor/execParallel.c | 14 +-
src/backend/executor/execProcnode.c | 46 ++++
src/backend/executor/nodeGather.c | 299 +++++++++++++++++++++++++
src/backend/nodes/copyfuncs.c | 25 +++
src/backend/nodes/outfuncs.c | 14 ++
src/backend/optimizer/path/costsize.c | 38 ++++
src/backend/optimizer/plan/createplan.c | 57 +++++
src/backend/optimizer/plan/setrefs.c | 1 +
src/backend/optimizer/plan/subselect.c | 1 +
src/backend/optimizer/util/pathnode.c | 26 +++
src/backend/utils/misc/guc.c | 30 +++
src/backend/utils/misc/postgresql.conf.sample | 3 +
src/include/executor/executor.h | 1 +
src/include/executor/nodeGather.h | 25 +++
src/include/nodes/execnodes.h | 16 ++
src/include/nodes/nodes.h | 3 +
src/include/nodes/plannodes.h | 11 +
src/include/nodes/relation.h | 13 ++
src/include/optimizer/cost.h | 7 +
src/include/optimizer/pathnode.h | 3 +
src/tools/pgindent/typedefs.list | 4 +
26 files changed, 709 insertions(+), 8 deletions(-)

Hi,

On 2015-09-30 23:29:30 +0000, Robert Haas wrote:
> Add a Gather executor node.
> ...
> src/backend/executor/execProcnode.c | 46 ++++

I just noticed that this added a new execProcnode dispatch routine, but
didn't add that to the file's header

* INTERFACE ROUTINES
* ExecInitNode - initialize a plan node and its subplans
* ExecProcNode - get a tuple by executing the plan node
* ExecEndNode - shut down a plan node and its subplans
*

Greetings,

Andres Freund

On Fri, Jul 21, 2017 at 7:06 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> On 2015-09-30 23:29:30 +0000, Robert Haas wrote:
>> Add a Gather executor node.
>> ...
>> src/backend/executor/execProcnode.c | 46 ++++
>
> I just noticed that this added a new execProcnode dispatch routine, but
> didn't add that to the file's header
>
> * INTERFACE ROUTINES
> * ExecInitNode - initialize a plan node and its subplans
> * ExecProcNode - get a tuple by executing the plan node
> * ExecEndNode - shut down a plan node and its subplans
> *

I think those top-of-file lists are just annoying, and would prefer to
rip them out entirely rather than spend time maintaining them. If you
want to see the list of interface routines, look in the header file.

heapam.c's header, for example, is missing try_relation_open,
relation_openrv_extended, heap_openrv_extended,
heap_beginscan_catalog, heap_beginscan_strat, heap_beginscan_bm,
heap_beginscan_sampling, heap_setscanlimits, heapgetpage,
heap_rescan_set_params, heap_parallelscan_estimate,
heap_parallelscan_initialize, heap_beginscan_parallel,
heap_hot_search_buffer, heap_hot_search, setLastTid,
GetBulkInsertState, FreeBulkInsertState, ReleaseBulkInsertState,
heap_finish_speculative, heap_abort_speculative, heap_lock_tuple,
heap_inplace_update, heap_tuple_needs_freeze,
heap_tuple_needs_eventual_freeze, simple_heap_insert,
simple_heap_update, simple_heap_delete, and heap_update_snapshot,
which means that if this comment was ever correct, it was more than a
decade ago.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas wrote:

> I think those top-of-file lists are just annoying, and would prefer to
> rip them out entirely rather than spend time maintaining them.

+1

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On July 21, 2017 5:34:32 PM GMT+02:00, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> wrote:
>Robert Haas wrote:
>
>> I think those top-of-file lists are just annoying, and would prefer
>to
>> rip them out entirely rather than spend time maintaining them.
>
>+1

I'm quite sympathetic to that view. But I think it's either them, or ripping out, not leaving bit rot.

Andres

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> writes:
> Robert Haas wrote:
>> I think those top-of-file lists are just annoying, and would prefer to
>> rip them out entirely rather than spend time maintaining them.

> +1

To the extent that they're just lists of function names, +1. Some of
them have some documentation in them, which you'd need to make sure
is duplicative or else copy it to an appropriate place.

regards, tom lane