Re: Documenting pglesslog

In thinking about how to communicate to users about reducing continuous
archiving storage requirements, I realized we don't mention pglesslog in
our official documentation.

The attached patch documents how to use pglesslog and gzip/gunzip to
reduce storage requirements. Comments?

Also, I assume pg_lesslog removes the padding we use to make all WAL
files 16MB, effectively doing the function of clearxlogtail too, right?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Sat, 2009-01-10 at 21:09 -0500, Bruce Momjian wrote:

> Comments?

If this is for backpatching, it makes sense. We should at least wait
until sync rep is accepted or rejected and docs written.

In general I don't think we should refer/link to other companies'
copyrighted materials in our documentation. That could cause
difficulties.

If you're going to do this, then I think you should go through the docs
and refer directly to many other commonly used tools that are also on
pg_foundry.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs wrote:
>
> On Sat, 2009-01-10 at 21:09 -0500, Bruce Momjian wrote:
>
> > Comments?
>
> If this is for backpatching, it makes sense. We should at least wait
> until sync rep is accepted or rejected and docs written.

No, it is not for backpatching.

> In general I don't think we should refer/link to other companies'
> copyrighted materials in our documentation. That could cause
> difficulties.

It is BSD licensed. I don't see any copyright issues:

http://pglesslog.projects.postgresql.org/

> If you're going to do this, then I think you should go through the docs
> and refer directly to many other commonly used tools that are also on
> pg_foundry.

I add items where they fit logically.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Sun, 2009-01-11 at 03:12 +0000, Simon Riggs wrote:
> On Sat, 2009-01-10 at 21:09 -0500, Bruce Momjian wrote:
>
> > Comments?
>
> If this is for backpatching, it makes sense. We should at least wait
> until sync rep is accepted or rejected and docs written.

Why? Even if sync rep is accepted, pglesslog would still be useful for
those who aren't using wal archiving right?

>
> In general I don't think we should refer/link to other companies'
> copyrighted materials in our documentation. That could cause
> difficulties.

What? That seems a bit odd. I see zero problem with linking to the page,
especially considering it is an open source project hosted on a
postgresql project service.

>
> If you're going to do this, then I think you should go through the docs
> and refer directly to many other commonly used tools that are also on
> pg_foundry.
>

Well have some more information would certainly be useful but in this
particular case I don't know of anything else on pgfoundry that would
actually help with the problem Bruce is trying to solve.

Joshua D. Drake

> --
> Simon Riggs www.2ndQuadrant.com
> PostgreSQL Training, Services and Support
>
>
--
PostgreSQL
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Sat, 2009-01-10 at 23:38 -0500, Bruce Momjian wrote:

> It is BSD licensed. I don't see any copyright issues:
>
> http://pglesslog.projects.postgresql.org/

A licence and copyright are different things. Why do we insist on
changing copyright on our code if it is unimportant?

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs wrote:
>
> On Sat, 2009-01-10 at 23:38 -0500, Bruce Momjian wrote:
>
> > It is BSD licensed. I don't see any copyright issues:
> >
> > http://pglesslog.projects.postgresql.org/
>
> A licence and copyright are different things. Why do we insist on
> changing copyright on our code if it is unimportant?

Because the BSD copyright has no enforcement, I don't think the
copyright holder is important --- look at all the companies that take
our code and use it in their commercial products. We rebrand code we
accept so it is clear who maintains it; note we take NetBSD code and
add our copyright name to theirs and distribute it as our own.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Sun, 2009-01-11 at 09:47 -0500, Bruce Momjian wrote:
> Simon Riggs wrote:
> >
> > On Sat, 2009-01-10 at 23:38 -0500, Bruce Momjian wrote:
> >
> > > It is BSD licensed. I don't see any copyright issues:
> > >
> > A licence and copyright are different things. Why do we insist on
> > changing copyright on our code if it is unimportant?
>
> Because the BSD copyright has no enforcement

AFAIK there is no such thing as a BSD copyright. There is Copyright and
there is a BSD licence, issued by the copyright holder.

In general, IMHO, I don't think it's a good direction to go in to
include links to works of other copyright holders.

Specifically, I have nothing but good to say about pglesslog and its
authors.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

> In general, IMHO, I don't think it's a good direction to go in to
> include links to works of other copyright holders.

I think it's a great idea. IMHO, one of the major selling points of
PostgreSQL is its awesome documentation. However, one of its
weaknesses is that contrib module, pgfoundry projects, etc. are often
not mentioned in the parts of the main documentation to which they
relate. While I certainly don't want to go in the direction of
telling people "Don't worry about the fact that we handle X poorly
because there is a 5-year old, unmaintained pgfoundry module that
fixes it", giving people references tools that the community thinks
are good and useful seems very helpful to me.

I am completely mystified as what linking "other copyright holders"
has to do with it. That seems to imply that you fear some sort of
legal entanglement, but I can't imagine what it could possibly be.
Admittedly, IANAL.

...Robert

Bruce Momjian wrote:
> In thinking about how to communicate to users about reducing continuous
> archiving storage requirements, I realized we don't mention pglesslog in
> our official documentation.
>
> The attached patch documents how to use pglesslog and gzip/gunzip to
> reduce storage requirements. Comments?
>
> Also, I assume pg_lesslog removes the padding we use to make all WAL
> files 16MB, effectively doing the function of clearxlogtail too, right?

Applied.

---------------------------------------------------------------------------

>
> --
> Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
> EnterpriseDB http://enterprisedb.com
>
> + If your life is a hard drive, Christ can be your backup. +

[ text/x-diff is unsupported, treating like TEXT/PLAIN ]

> Index: doc/src/sgml/backup.sgml
> ===================================================================
> RCS file: /cvsroot/pgsql/doc/src/sgml/backup.sgml,v
> retrieving revision 2.121
> diff -c -c -r2.121 backup.sgml
> *** doc/src/sgml/backup.sgml 9 Nov 2008 17:51:15 -0000 2.121
> --- doc/src/sgml/backup.sgml 11 Jan 2009 01:41:12 -0000
> ***************
> *** 1337,1342 ****
> --- 1337,1359 ----
> WAL files are part of the same <application>tar</> file.
> Please remember to add error handling to your backup scripts.
> </para>
> +
> + <para>
> + If archive storage size is a concern, use <application>pg_compresslog</>,
> + <ulink url="http://pglesslog.projects.postgresql.org"></ulink>, to
> + remove unnecessary <xref linkend="guc-full-page-writes"> and trailing
> + space from the WAL files. You can then use
> + <application>gzip</application> to further compress the output of
> + <application>pg_compresslog</>:
> + <programlisting>
> + archive_command = 'pg_compresslog %p - | gzip &gt; /var/lib/pgsql/archive/%f'
> + </programlisting>
> + You will then need to use <application>gunzip</> and
> + <application>pg_decompresslog</> during recovery:
> + <programlisting>
> + restore_command = 'gunzip &lt; /mnt/server/archivedir/%f | pg_decompresslog - %p'
> + </programlisting>
> + </para>
> </sect3>
>
> <sect3 id="backup-scripts">

>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Hi,

I have no intention to make pglesslog to conflict to PostgreSQL
license. Any advice is welcome to make pglesslog available without
any license concern.

I've a question and ideas.

Bruce's modification directly points to my pgfoundry page. I'm not
sure what it means. Does it mean that I have to maintain the page for
a while? If pglesslog helps for future releases, can it be a part of
PostgreSQL release, as contrib module so that all the documentation in
pgfoundry (although very simple) is included in the release material?

As many hackers know, I've posted another code to speedup PITR after
slipping FPW, which does work with 8.3 as external module
(pg_readahead). I'm now working to work this with synchronous
replication. Maybe it's a good idea to use pglesslog with
pg_readahead. Although I'm not sure if pg_readahead integration
with synchronous replication will be done within 8.4 development
period, I'm quite ready to post pg_readahead for 8.4 sililar to that
for 8.3, which also could be in contrib module.

Looking forward to inputs.

2009/1/12 Simon Riggs <simon(at)2ndquadrant(dot)com>:
>
> On Sun, 2009-01-11 at 09:47 -0500, Bruce Momjian wrote:
>> Simon Riggs wrote:
>> >
>> > On Sat, 2009-01-10 at 23:38 -0500, Bruce Momjian wrote:
>> >
>> > > It is BSD licensed. I don't see any copyright issues:
>> > >
>> > A licence and copyright are different things. Why do we insist on
>> > changing copyright on our code if it is unimportant?
>>
>> Because the BSD copyright has no enforcement
>
> AFAIK there is no such thing as a BSD copyright. There is Copyright and
> there is a BSD licence, issued by the copyright holder.
>
> In general, IMHO, I don't think it's a good direction to go in to
> include links to works of other copyright holders.
>
> Specifically, I have nothing but good to say about pglesslog and its
> authors.
>
> --
> Simon Riggs www.2ndQuadrant.com
> PostgreSQL Training, Services and Support
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

--
------
Koichi Suzuki

On Tue, 2009-01-13 at 13:21 +0900, Koichi Suzuki wrote:

> I have no intention to make pglesslog to conflict to PostgreSQL
> license. Any advice is welcome to make pglesslog available without
> any license concern.

I understand, no part of my comments were against you or your work.

> I've a question and ideas.
>
> Bruce's modification directly points to my pgfoundry page. I'm not
> sure what it means. Does it mean that I have to maintain the page for
> a while? If pglesslog helps for future releases, can it be a part of
> PostgreSQL release, as contrib module so that all the documentation in
> pgfoundry (although very simple) is included in the release material?

I think it would be better to create a Wiki page that is directly
controlled by the project, which describes additions to PITR (or other
aspects of the project) and contains links. I think everyone accepts
that the Wiki can have off-project links.

That way people can submit their work without needing to make
off-project links permanent from the docs. If people then change their
site content in future we can more easily change the link.

For example, Josh can make contributions there as well.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Koichi Suzuki wrote:
> Hi,
>
> I have no intention to make pglesslog to conflict to PostgreSQL
> license. Any advice is welcome to make pglesslog available without
> any license concern.

I certainly have no concerns.

> I've a question and ideas.
>
> Bruce's modification directly points to my pgfoundry page. I'm not
> sure what it means. Does it mean that I have to maintain the page for
> a while? If pglesslog helps for future releases, can it be a part of
> PostgreSQL release, as contrib module so that all the documentation in
> pgfoundry (although very simple) is included in the release material?

I think eventually we should put pglesslog into /contrib, and if we ever
do that, we would update your web page. I have not heard any mention of
it being moved into /contrib for 8.4 though.

If you would like me to point to another URL, please let me know.

I think there is definately demand for pglesslog because not only does
it truncate dead space from the WAL file, it also removes full page
write images, and is best done in archive_command, and hence externally
like your tool does.

> As many hackers know, I've posted another code to speedup PITR after
> slipping FPW, which does work with 8.3 as external module
> (pg_readahead). I'm now working to work this with synchronous
> replication. Maybe it's a good idea to use pglesslog with
> pg_readahead. Although I'm not sure if pg_readahead integration
> with synchronous replication will be done within 8.4 development
> period, I'm quite ready to post pg_readahead for 8.4 sililar to that
> for 8.3, which also could be in contrib module.

Sorry, I don't know enough about pg_readahead.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Pg_readahead is a tool to prefetch data pages before redoing, based on
the contents of archive/active WAL segment. For 8.3 and 8.4 without
sync.rep, this works together with restore_command. Pg_readahead
analyze WAL segment, schedule and issue posix_fadvise() to prefetch
data pages quickly before redoing.

Discussions and materials will be found at

http://archives.postgresql.org/pgsql-hackers/2008-10/msg01372.php

So far, external command implemantation speeds up PITR up to six
times! Therefore, overall recovery time can be a little longer than
that with FPW.

2009/1/14 Bruce Momjian <bruce(at)momjian(dot)us>:
> Koichi Suzuki wrote:
>> Hi,
>>
>> I have no intention to make pglesslog to conflict to PostgreSQL
>> license. Any advice is welcome to make pglesslog available without
>> any license concern.
>
> I certainly have no concerns.
>
>> I've a question and ideas.
>>
>> Bruce's modification directly points to my pgfoundry page. I'm not
>> sure what it means. Does it mean that I have to maintain the page for
>> a while? If pglesslog helps for future releases, can it be a part of
>> PostgreSQL release, as contrib module so that all the documentation in
>> pgfoundry (although very simple) is included in the release material?
>
> I think eventually we should put pglesslog into /contrib, and if we ever
> do that, we would update your web page. I have not heard any mention of
> it being moved into /contrib for 8.4 though.
>
> If you would like me to point to another URL, please let me know.
>
> I think there is definately demand for pglesslog because not only does
> it truncate dead space from the WAL file, it also removes full page
> write images, and is best done in archive_command, and hence externally
> like your tool does.
>
>> As many hackers know, I've posted another code to speedup PITR after
>> slipping FPW, which does work with 8.3 as external module
>> (pg_readahead). I'm now working to work this with synchronous
>> replication. Maybe it's a good idea to use pglesslog with
>> pg_readahead. Although I'm not sure if pg_readahead integration
>> with synchronous replication will be done within 8.4 development
>> period, I'm quite ready to post pg_readahead for 8.4 sililar to that
>> for 8.3, which also could be in contrib module.
>
> Sorry, I don't know enough about pg_readahead.
>
> --
> Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
> EnterpriseDB http://enterprisedb.com
>
> + If your life is a hard drive, Christ can be your backup. +
>

--
------
Koichi Suzuki

Koichi Suzuki wrote:
> Pg_readahead is a tool to prefetch data pages before redoing, based on
> the contents of archive/active WAL segment. For 8.3 and 8.4 without
> sync.rep, this works together with restore_command. Pg_readahead
> analyze WAL segment, schedule and issue posix_fadvise() to prefetch
> data pages quickly before redoing.
>
> Discussions and materials will be found at
>
> http://archives.postgresql.org/pgsql-hackers/2008-10/msg01372.php
>
> So far, external command implemantation speeds up PITR up to six
> times! Therefore, overall recovery time can be a little longer than
> that with FPW.

Now that 8.4 is using fsync, sounds like something that should be
integrated into the core code, rather than as a /contrib.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Pg_readahead uses posix_fadvise, which is included in Greg's patch and
I've already posted pg_readahead patch integrated into the core.

Integration with snc.rep. will be a separate patch which will be
posted in a couple of days.

2009/1/14 Bruce Momjian <bruce(at)momjian(dot)us>:
> Koichi Suzuki wrote:
>> Pg_readahead is a tool to prefetch data pages before redoing, based on
>> the contents of archive/active WAL segment. For 8.3 and 8.4 without
>> sync.rep, this works together with restore_command. Pg_readahead
>> analyze WAL segment, schedule and issue posix_fadvise() to prefetch
>> data pages quickly before redoing.
>>
>> Discussions and materials will be found at
>>
>> http://archives.postgresql.org/pgsql-hackers/2008-10/msg01372.php
>>
>> So far, external command implemantation speeds up PITR up to six
>> times! Therefore, overall recovery time can be a little longer than
>> that with FPW.
>
> Now that 8.4 is using fsync, sounds like something that should be
> integrated into the core code, rather than as a /contrib.
>
> --
> Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
> EnterpriseDB http://enterprisedb.com
>
> + If your life is a hard drive, Christ can be your backup. +
>

--
------
Koichi Suzuki