Re: pg_restore -j <nothing>

Hi,

I just noticed (!) that Make accepts an argument-less -j option, which
it takes to mean "use as many parallel jobs as possible". As far as I
see in our pg_restore code, we don't even accept an argumentless -j
option; was this deviation from the Make precedent on purpose, or were
we just not following Make at all on this?

I have to admit that I'm not really sure whether this kind of usage
would be a reasonable thing for pg_restore to support.

(Even if this was a good idea, I'm not suggesting that it be implemented
for 8.4. But if it is, then maybe it deserves a TODO entry.)

Thoughts?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera wrote:
> Hi,
>
> I just noticed (!) that Make accepts an argument-less -j option, which
> it takes to mean "use as many parallel jobs as possible". As far as I
> see in our pg_restore code, we don't even accept an argumentless -j
> option; was this deviation from the Make precedent on purpose, or were
> we just not following Make at all on this?
>
> I have to admit that I'm not really sure whether this kind of usage
> would be a reasonable thing for pg_restore to support.
>
> (Even if this was a good idea, I'm not suggesting that it be implemented
> for 8.4. But if it is, then maybe it deserves a TODO entry.)

An unlimited pg_restore -j seems pretty scary.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Alvaro Herrera wrote:
> Hi,
>
> I just noticed (!) that Make accepts an argument-less -j option, which
> it takes to mean "use as many parallel jobs as possible". As far as I
> see in our pg_restore code, we don't even accept an argumentless -j
> option; was this deviation from the Make precedent on purpose, or were
> we just not following Make at all on this?
>
> I have to admit that I'm not really sure whether this kind of usage
> would be a reasonable thing for pg_restore to support.
>
> (Even if this was a good idea, I'm not suggesting that it be implemented
> for 8.4. But if it is, then maybe it deserves a TODO entry.)
>
> Thoughts?
>
>

There was no intention to follow Make.

And I think it's far far too early to be planning "improvements" of this
kind. We need to see how it gets used in the field.

cheers

andrew

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Alvaro Herrera wrote:
>> I just noticed (!) that Make accepts an argument-less -j option, which
>> it takes to mean "use as many parallel jobs as possible".

> An unlimited pg_restore -j seems pretty scary.

Yeah. Even if Make has a sane way to estimate how many jobs it should
use, I'm not sure that pg_restore does. (The most obvious heuristic
for Make is to try to find out how many CPUs there are --- but at
least it's running on the same machine it's going to be eating CPU
on. pg_restore can't assume that.)

regards, tom lane

On Thursday 23 April 2009 01:26:04 Alvaro Herrera wrote:
> I just noticed (!) that Make accepts an argument-less -j option, which
> it takes to mean "use as many parallel jobs as possible". As far as I
> see in our pg_restore code, we don't even accept an argumentless -j
> option; was this deviation from the Make precedent on purpose, or were
> we just not following Make at all on this?

There was likely no strong intention to follow make on this. A small problem
would be that getopt doesn't portably support single-letter options with
optional arguments.

The main problem, however, is that make -j is pretty useless and dangerous.
Using it on a large parallel-make-safe project can easily lock up (thrash)
your machine. make -j together with -l (--load-average) is kind of useful, I
guess, but exactly how "load average" translated to a PostgreSQL database
system is to be determined.

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> Yeah. Even if Make has a sane way to estimate how many jobs it should
> use, I'm not sure that pg_restore does. (The most obvious heuristic
> for Make is to try to find out how many CPUs there are --- but at
> least it's running on the same machine it's going to be eating CPU
> on. pg_restore can't assume that.)

I'm not sure if I'd consider it 'sane', but make basically uses the
dependency information, if a job can be run based on its dependency
requirements, then it's started. For small projects, this isn't
necessairly terrible, but it's not something I would generally
recommend.

I don't see any reasonable implementation, or justification, for
supporting something like that in pg_restore.

Thanks,

Stephen