Re: parallel regression test failure

I am seeing the following parallel regression test failures. Any idea
on the cause?

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> I am seeing the following parallel regression test failures. Any idea
> on the cause?

I don't see it here, on either of two different architectures. Maybe
you need a make distclean and rebuild?

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > I am seeing the following parallel regression test failures. Any idea
> > on the cause?
>
> I don't see it here, on either of two different architectures. Maybe
> you need a make distclean and rebuild?

I do (I run tools/pgtest), and see the failure regularly. It is a
dual-cpu Xeon machine. I run it every night and it fails 25% of the
time.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

On Fri, 25 Jul 2003 19:57:04 -0400
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> said something like:

> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > I am seeing the following parallel regression test failures. Any
> > idea on the cause?
>
> I don't see it here, on either of two different architectures. Maybe
> you need a make distclean and rebuild?
>

I was just able to get some problems on my dual Athlon machine
(after about 10 runs) with a clean cvs download.

Linux thunder.mshome.net 2.4.21-0.13_test #35 SMP Wed Apr 9 07:29:10 MDT
2003 i686 unknown unknown GNU/Linux

gcc (GCC) 3.2.2 (Mandrake Linux 9.1 3.2.2-3mdk)

<./configure --with-pgport=5433 --prefix=/usr/local/pgsql_cvs>
<sh src/tools/pgtest>
<sh src/tools/pgtest -n>

*** ./expected/triggers.out Thu Jul 24 11:52:50 2003
--- ./results/triggers.out Fri Jul 25 21:20:34 2003
***************
*** 92,97 ****
--- 92,98 ----
DROP TABLE pkeys;
DROP TABLE fkeys;
DROP TABLE fkeys2;
+ ERROR: could not open relation with OID 119498
-- -- I've disabled the funny_dup17 test because the new semantics
-- -- of AFTER ROW triggers, which get now fired at the end of a
-- -- query always, cause funny_dup17 to enter an endless loop.

======================================================================

*** ./expected/sanity_check.out Wed May 28 10:04:02 2003
--- ./results/sanity_check.out Fri Jul 25 21:20:37 2003
***************
*** 15,20 ****
--- 15,21 ----
bt_name_heap | t
bt_txt_heap | t
fast_emp4000 | t
+ fkeys2 | t
func_index_heap | t
hash_f8_heap | t
hash_i4_heap | t
***************
*** 62,68 ****
shighway | t
tenk1 | t
tenk2 | t
! (52 rows)

--
-- another sanity check: every system catalog that has OIDs should
have--- 63,69 ----
shighway | t
tenk1 | t
tenk2 | t
! (53 rows)

--
-- another sanity check: every system catalog that has OIDs should
have

======================================================================

*** ./expected/misc.out Fri Jul 25 21:14:51 2003
--- ./results/misc.out Fri Jul 25 21:20:39 2003
***************
*** 598,603 ****
--- 598,604 ----
equipment_r
f_star
fast_emp4000
+ fkeys2
float4_tbl
float8_tbl
func_index_heap
***************
*** 660,666 ****
toyemp
varchar_tbl
xacttest
! (96 rows)

--SELECT name(equipment(hobby_construct(text 'skywalking', text
'mer'))) AS equip_name; SELECT hobbies_by_name('basketball');
--- 661,667 ----
toyemp
varchar_tbl
xacttest
! (97 rows)

--SELECT name(equipment(hobby_construct(text 'skywalking', text
'mer'))) AS equip_name; SELECT hobbies_by_name('basketball');

======================================================================

--
21:23:44 up 8 days, 1:24, 2 users, load average: 0.11, 1.04, 1.31

On Fri, 25 Jul 2003 19:57:04 -0400
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> said something like:

> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > I am seeing the following parallel regression test failures. Any
> > idea on the cause?
>
> I don't see it here, on either of two different architectures. Maybe
> you need a make distclean and rebuild?
>

And another failure:

*** ./expected/constraints.out Fri Jul 25 21:14:51 2003
--- ./results/constraints.out Fri Jul 25 21:34:09 2003
***************
*** 212,244 ****
DROP SEQUENCE INSERT_SEQ;
CREATE SEQUENCE INSERT_SEQ START 4;
CREATE TABLE tmp (xd INT, yd TEXT, zd INT);
INSERT INTO tmp VALUES (null, 'Y', null);
INSERT INTO tmp VALUES (5, '!check failed', null);
INSERT INTO tmp VALUES (null, 'try again', null);
INSERT INTO INSERT_TBL(y) select yd from tmp;
SELECT '' AS three, * FROM INSERT_TBL;
three | x | y | z
! -------+---+---------------+----
! | 4 | Y | -4
! | 5 | !check failed | -5
! | 6 | try again | -6
! (3 rows)

INSERT INTO INSERT_TBL SELECT * FROM tmp WHERE yd = 'try again';
INSERT INTO INSERT_TBL(y,z) SELECT yd, -7 FROM tmp WHERE yd = 'try
again'; INSERT INTO INSERT_TBL(y,z) SELECT yd, -8 FROM tmp WHERE yd =
'try again';! ERROR: new row for relation "insert_tbl" violates CHECK
constraint "insert_con" SELECT '' AS four, * FROM INSERT_TBL;
four | x | y | z
! ------+---+---------------+----
! | 4 | Y | -4
! | 5 | !check failed | -5
! | 6 | try again | -6
! | | try again |
! | 7 | try again | -7
! (5 rows)

DROP TABLE tmp;
--
-- Check constraints on UPDATE
--
--- 212,244 ----
DROP SEQUENCE INSERT_SEQ;
CREATE SEQUENCE INSERT_SEQ START 4;
CREATE TABLE tmp (xd INT, yd TEXT, zd INT);
+ ERROR: relation 126260 deleted while still in use
INSERT INTO tmp VALUES (null, 'Y', null);
+ ERROR: relation "tmp" does not exist
INSERT INTO tmp VALUES (5, '!check failed', null);
+ ERROR: relation "tmp" does not exist
INSERT INTO tmp VALUES (null, 'try again', null);
+ ERROR: relation "tmp" does not exist
INSERT INTO INSERT_TBL(y) select yd from tmp;
+ ERROR: relation "tmp" does not exist
SELECT '' AS three, * FROM INSERT_TBL;
three | x | y | z
! -------+---+---+---
! (0 rows)

INSERT INTO INSERT_TBL SELECT * FROM tmp WHERE yd = 'try again';
+ ERROR: relation "tmp" does not exist
INSERT INTO INSERT_TBL(y,z) SELECT yd, -7 FROM tmp WHERE yd = 'try
again';+ ERROR: relation "tmp" does not exist
INSERT INTO INSERT_TBL(y,z) SELECT yd, -8 FROM tmp WHERE yd = 'try
again';! ERROR: relation "tmp" does not exist
SELECT '' AS four, * FROM INSERT_TBL;
four | x | y | z
! ------+---+---+---
! (0 rows)

DROP TABLE tmp;
+ ERROR: table "tmp" does not exist
--
-- Check constraints on UPDATE
--
***************
*** 246,261 ****
UPDATE INSERT_TBL SET x = 6 WHERE x = 6;
UPDATE INSERT_TBL SET x = -z, z = -x;
UPDATE INSERT_TBL SET x = z, z = x;
- ERROR: new row for relation "insert_tbl" violates CHECK constraint
"insert_con" SELECT * FROM INSERT_TBL;
x | y | z
! ---+---------------+----
! 4 | Y | -4
! | try again |
! 7 | try again | -7
! 5 | !check failed |
! 6 | try again | -6
! (5 rows)

-- DROP TABLE INSERT_TBL;
--
--- 246,255 ----
UPDATE INSERT_TBL SET x = 6 WHERE x = 6;
UPDATE INSERT_TBL SET x = -z, z = -x;
UPDATE INSERT_TBL SET x = z, z = x;
SELECT * FROM INSERT_TBL;
x | y | z
! ---+---+---
! (0 rows)

-- DROP TABLE INSERT_TBL;
--

======================================================================

--
21:34:48 up 8 days, 1:35, 2 users, load average: 0.89, 0.65, 0.85

On Fri, 25 Jul 2003 19:57:04 -0400
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> said something like:

> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > I am seeing the following parallel regression test failures. Any
> > idea on the cause?
>
> I don't see it here, on either of two different architectures. Maybe
> you need a make distclean and rebuild?
>

I've attached a little Perl script which runs pgtest over and over
(with -n option), checking for failures and saving the output
(runX.out) and the diffs (failX.diffs) in /tmp for each failing run.

Run it from the top level (as you would pgtest).

Later,
Rob

--
22:25:11 up 8 days, 2:26, 2 users, load average: 2.40, 1.61, 1.57

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> I run it every night and it fails 25% of the time.

When did you start seeing the problem?

I just wasted an hour running eighty-some iterations of "make check"
on two different machines/OSes/architectures. Zero failures. I also
eyeballed recent changes in the relcache/catcache area, which seems to
be what's unhappy, without finding anything.

I think it's up to yunz as are seeing misbehavior to roll up your
sleeves and debug the problem. There's nothing more I can do.

regards, tom lane

Let me get the patch queue applied, then use CVS to backtrack and find
the date it started failing. I think you need a dual cpu machine to see
the failures.

---------------------------------------------------------------------------

Tom Lane wrote:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > I run it every night and it fails 25% of the time.
>
> When did you start seeing the problem?
>
> I just wasted an hour running eighty-some iterations of "make check"
> on two different machines/OSes/architectures. Zero failures. I also
> eyeballed recent changes in the relcache/catcache area, which seems to
> be what's unhappy, without finding anything.
>
> I think it's up to yunz as are seeing misbehavior to roll up your
> sleeves and debug the problem. There's nothing more I can do.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

On Sat, 26 Jul 2003 01:00:46 -0400
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> said something like:

> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > I run it every night and it fails 25% of the time.
>
> When did you start seeing the problem?
>
> I just wasted an hour running eighty-some iterations of "make check"
> on two different machines/OSes/architectures. Zero failures. I also
> eyeballed recent changes in the relcache/catcache area, which seems to
> be what's unhappy, without finding anything.
>
> I think it's up to yunz as are seeing misbehavior to roll up your
> sleeves and debug the problem. There's nothing more I can do.
>

Any suggestions for those of us who are not pg developers how I might
help figure out what's up?

1 of 25 - failed 0 (0%)
2 of 25 - failed 0 (0%)
3 of 25 - failed 0 (0%)
4 of 25 - failed 0 (0%)
5 of 25 - failed 0 (0%)
6 of 25 - failed 0 (0%)
7 of 25 - failed 0 (0%)
8 of 25 - failed 0 (0%)
9 of 25 - failed 0 (0%)
10 of 25 - failed 0 (0%)
11 of 25 - failed 1 (9%)
12 of 25 - failed 2 (17%)
13 of 25 - failed 2 (15%)
14 of 25 - failed 2 (14%
15 of 25 - failed 3 (20%)
16 of 25 - failed 3 (19%)
17 of 25 - failed 3 (18%)
18 of 25 - failed 4 (22%)
19 of 25 - failed 4 (21%)
20 of 25 - failed 4 (20%)
21 of 25 - failed 5 (24%)
22 of 25 - failed 6 (27%)
23 of 25 - failed 6 (26%)
24 of 25 - failed 7 (29%)
25 of 25 - failed 8 (32%)
constraints failed 1 times
cluster failed 1 times
foreign_key failed 1 times
misc failed 6 times
sanity_check failed 3 times
inherit failed 2 times
triggers failed 4 times

--
08:21:18 up 8 days, 12:22, 2 users, load average: 0.08, 0.65, 1.58

I am going to use cvs -d to pull an older CVS and see if that fails, so
we can track down the date it started failing.

---------------------------------------------------------------------------

Robert Creager wrote:
-- Start of PGP signed section.
>
> On Sat, 26 Jul 2003 01:00:46 -0400
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> said something like:
>
> > Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > > I run it every night and it fails 25% of the time.
> >
> > When did you start seeing the problem?
> >
> > I just wasted an hour running eighty-some iterations of "make check"
> > on two different machines/OSes/architectures. Zero failures. I also
> > eyeballed recent changes in the relcache/catcache area, which seems to
> > be what's unhappy, without finding anything.
> >
> > I think it's up to yunz as are seeing misbehavior to roll up your
> > sleeves and debug the problem. There's nothing more I can do.
> >
>
> Any suggestions for those of us who are not pg developers how I might
> help figure out what's up?
>
> 1 of 25 - failed 0 (0%)
> 2 of 25 - failed 0 (0%)
> 3 of 25 - failed 0 (0%)
> 4 of 25 - failed 0 (0%)
> 5 of 25 - failed 0 (0%)
> 6 of 25 - failed 0 (0%)
> 7 of 25 - failed 0 (0%)
> 8 of 25 - failed 0 (0%)
> 9 of 25 - failed 0 (0%)
> 10 of 25 - failed 0 (0%)
> 11 of 25 - failed 1 (9%)
> 12 of 25 - failed 2 (17%)
> 13 of 25 - failed 2 (15%)
> 14 of 25 - failed 2 (14%
> 15 of 25 - failed 3 (20%)
> 16 of 25 - failed 3 (19%)
> 17 of 25 - failed 3 (18%)
> 18 of 25 - failed 4 (22%)
> 19 of 25 - failed 4 (21%)
> 20 of 25 - failed 4 (20%)
> 21 of 25 - failed 5 (24%)
> 22 of 25 - failed 6 (27%)
> 23 of 25 - failed 6 (26%)
> 24 of 25 - failed 7 (29%)
> 25 of 25 - failed 8 (32%)
> constraints failed 1 times
> cluster failed 1 times
> foreign_key failed 1 times
> misc failed 6 times
> sanity_check failed 3 times
> inherit failed 2 times
> triggers failed 4 times
>
>
> --
> 08:21:18 up 8 days, 12:22, 2 users, load average: 0.08, 0.65, 1.58
-- End of PGP section, PGP failed!

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

If you would like to do the cvs -d testing yourself instead of me, let
me know. It will take me a few hours to get to it anyway.

---------------------------------------------------------------------------

Robert Creager wrote:
-- Start of PGP signed section.
>
> On Sat, 26 Jul 2003 01:00:46 -0400
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> said something like:
>
> > Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > > I run it every night and it fails 25% of the time.
> >
> > When did you start seeing the problem?
> >
> > I just wasted an hour running eighty-some iterations of "make check"
> > on two different machines/OSes/architectures. Zero failures. I also
> > eyeballed recent changes in the relcache/catcache area, which seems to
> > be what's unhappy, without finding anything.
> >
> > I think it's up to yunz as are seeing misbehavior to roll up your
> > sleeves and debug the problem. There's nothing more I can do.
> >
>
> Any suggestions for those of us who are not pg developers how I might
> help figure out what's up?
>
> 1 of 25 - failed 0 (0%)
> 2 of 25 - failed 0 (0%)
> 3 of 25 - failed 0 (0%)
> 4 of 25 - failed 0 (0%)
> 5 of 25 - failed 0 (0%)
> 6 of 25 - failed 0 (0%)
> 7 of 25 - failed 0 (0%)
> 8 of 25 - failed 0 (0%)
> 9 of 25 - failed 0 (0%)
> 10 of 25 - failed 0 (0%)
> 11 of 25 - failed 1 (9%)
> 12 of 25 - failed 2 (17%)
> 13 of 25 - failed 2 (15%)
> 14 of 25 - failed 2 (14%
> 15 of 25 - failed 3 (20%)
> 16 of 25 - failed 3 (19%)
> 17 of 25 - failed 3 (18%)
> 18 of 25 - failed 4 (22%)
> 19 of 25 - failed 4 (21%)
> 20 of 25 - failed 4 (20%)
> 21 of 25 - failed 5 (24%)
> 22 of 25 - failed 6 (27%)
> 23 of 25 - failed 6 (26%)
> 24 of 25 - failed 7 (29%)
> 25 of 25 - failed 8 (32%)
> constraints failed 1 times
> cluster failed 1 times
> foreign_key failed 1 times
> misc failed 6 times
> sanity_check failed 3 times
> inherit failed 2 times
> triggers failed 4 times
>
>
> --
> 08:21:18 up 8 days, 12:22, 2 users, load average: 0.08, 0.65, 1.58
-- End of PGP section, PGP failed!

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

On Sat, 26 Jul 2003 10:47:12 -0400 (EDT)
Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> said something like:

>
> If you would like to do the cvs -d testing yourself instead of me, let
> me know. It will take me a few hours to get to it anyway.
>

I will start doing pulling down old versions (once I figure out the -d
syntax). Do you recall how long you may of been seeing this?

Thanks,
Rob

--
08:54:59 up 8 days, 12:55, 2 users, load average: 2.38, 1.12, 1.14

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> I think you need a dual cpu machine to see the failures.

I was wondering about that myself, but we shouldn't fixate on that
assumption without more evidence. There could be some other factor
explaining why I can't reproduce it. A couple of questions for both
of you:
- what configure options are you using?
- can you reproduce the problem with serial tests (make installcheck)?
- exactly how repeatable is it --- when it fails, is it always at the
same places, or do the failures move around?

It would also be good to find out exactly where the failures are coming
from. Please try running the tests with LOG_ERROR_VERBOSITY set to
VERBOSE (probably the easiest way to hack this in make check's temp
installation is to modify src/backend/utils/misc/postgresql.conf.sample).
Then the postmaster log file created by make check will show the elog
calls' locations.

regards, tom lane

On Sat, 26 Jul 2003 10:47:12 -0400 (EDT)
Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> said something like:

>
> If you would like to do the cvs -d testing yourself instead of me, let
> me know. It will take me a few hours to get to it anyway.
>

Just to make sure I've got this right:

cvs update -D yyyy-mm-dd
make maintainer-clean
./configure
make
test

--
09:05:56 up 8 days, 13:06, 2 users, load average: 2.59, 2.90, 2.14

Robert Creager wrote:
-- Start of PGP signed section.
> On Sat, 26 Jul 2003 10:47:12 -0400 (EDT)
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> said something like:
>
> >
> > If you would like to do the cvs -d testing yourself instead of me, let
> > me know. It will take me a few hours to get to it anyway.
> >
>
> I will start doing pulling down old versions (once I figure out the -d
> syntax). Do you recall how long you may of been seeing this?

I think you just take a CVS checkout and to:

cvs update -D '2003-05-01 00:00:00 GMT' pgsql

and keep changing the dates to find the date it started breaking.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Yep, I think that is it, though the last one is pgtest or whatever you
are using for testing.

---------------------------------------------------------------------------

Robert Creager wrote:
-- Start of PGP signed section.
> On Sat, 26 Jul 2003 10:47:12 -0400 (EDT)
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> said something like:
>
> >
> > If you would like to do the cvs -d testing yourself instead of me, let
> > me know. It will take me a few hours to get to it anyway.
> >
>
> Just to make sure I've got this right:
>
> cvs update -D yyyy-mm-dd
> make maintainer-clean
> ./configure
> make
> test
>
> --
> 09:05:56 up 8 days, 13:06, 2 users, load average: 2.59, 2.90, 2.14
-- End of PGP section, PGP failed!

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Robert Creager wrote:
-- Start of PGP signed section.
> On Sat, 26 Jul 2003 10:47:12 -0400 (EDT)
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> said something like:
>
> >
> > If you would like to do the cvs -d testing yourself instead of me, let
> > me know. It will take me a few hours to get to it anyway.
> >
>
> I will start doing pulling down old versions (once I figure out the -d
> syntax). Do you recall how long you may of been seeing this?

Since it is random, I hadn't noticed when it started, and originally
suspected my hardware I recently upgraded my hardware, around May 1, I
think.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Tom Lane wrote:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > I think you need a dual cpu machine to see the failures.
>
> I was wondering about that myself, but we shouldn't fixate on that
> assumption without more evidence. There could be some other factor
> explaining why I can't reproduce it. A couple of questions for both
> of you:
> - what configure options are you using?

configure \
--with-x \
--with-threads \
--with-tcl \
--with-perl \
--with-python \
--enable-pltcl-unknown \
--with-tclconfig=/u/lib \
--with-tkconfig=/u/lib \
--enable-cassert \
--with-includes="/usr/local/include/readline /usr/contrib/include" \
--with-libraries="/usr/local/lib /usr/contrib/lib" \
--enable-locale \
--enable-multibyte \
--with-recode \
--with-openssl

> - can you reproduce the problem with serial tests (make installcheck)?

No, I have never seen a serial failure, and when I get a paralell
failure, I run the serial to make sure it is just the paralell test, and
serial always passes.

> - exactly how repeatable is it --- when it fails, is it always at the
> same places, or do the failures move around?

No, different, as reported by Robert, but it usually has to do with the
contraint, trigger, and sanity tests. I assume we just had a dependency
in the paralell regression tests and we just need to do an adjustment,
but looking at the diffs more closely, I see it is more serious.

> It would also be good to find out exactly where the failures are coming
> from. Please try running the tests with LOG_ERROR_VERBOSITY set to
> VERBOSE (probably the easiest way to hack this in make check's temp
> installation is to modify src/backend/utils/misc/postgresql.conf.sample).
> Then the postmaster log file created by make check will show the elog
> calls' locations.

OK.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Robert Creager <Robert_Creager(at)LogicalChaos(dot)org> writes:
> Just to make sure I've got this right:

> cvs update -D yyyy-mm-dd
> make maintainer-clean
> ./configure
> make
> test

I'd do the "make maintainer-clean" before cvs update'ing, but otherwise
probably right. Watch the output the first couple times and make sure
cvs is actually willing to replace files in both the forward and
backward directions.

regards, tom lane

./configure --with-pgport=5433 --prefix=/usr/local/pgsql_cvs

The failure moves around (out of 25 tests):

constraints failed 1 times
cluster failed 1 times
foreign_key failed 1 times
misc failed 6 times
sanity_check failed 3 times
inherit failed 2 times
triggers failed 4 times

Have not tried install check yet.

On Sat, 26 Jul 2003 11:06:21 -0400
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> said something like:

> - what configure options are you using?
> - can you reproduce the problem with serial tests (make
> installcheck)?- exactly how repeatable is it --- when it fails, is
> it always at the
> same places, or do the failures move around?
>

--
09:22:25 up 8 days, 13:23, 2 users, load average: 1.36, 1.26, 1.70

On Sat, 26 Jul 2003 11:22:21 -0400
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> said something like:

> Robert Creager <Robert_Creager(at)LogicalChaos(dot)org> writes:
> > Just to make sure I've got this right:
>
> > cvs update -D yyyy-mm-dd
> > make maintainer-clean
> > ./configure
> > make
> > test
>
> I'd do the "make maintainer-clean" before cvs update'ing, but
> otherwise probably right. Watch the output the first couple times and
> make sure cvs is actually willing to replace files in both the forward
> and backward directions.
>

Yeah, and yeah, it just removed src/tools/pgtest when I went back to
April...

--
09:36:18 up 8 days, 13:37, 2 users, load average: 0.08, 0.86, 1.54

Tom Lane wrote:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > I think you need a dual cpu machine to see the failures.
>
> I was wondering about that myself, but we shouldn't fixate on that
> assumption without more evidence. There could be some other factor
> explaining why I can't reproduce it. A couple of questions for both
> of you:
> - what configure options are you using?
> - can you reproduce the problem with serial tests (make installcheck)?
> - exactly how repeatable is it --- when it fails, is it always at the
> same places, or do the failures move around?
>
> It would also be good to find out exactly where the failures are coming
> from. Please try running the tests with LOG_ERROR_VERBOSITY set to
> VERBOSE (probably the easiest way to hack this in make check's temp
> installation is to modify src/backend/utils/misc/postgresql.conf.sample).
> Then the postmaster log file created by make check will show the elog
> calls' locations.

OK, I got a failure with verbose output. Error was:

*** ./expected/triggers.out Fri Jul 25 12:38:34 2003
--- ./results/triggers.out Sat Jul 26 12:52:02 2003
***************
*** 66,71 ****
--- 66,72 ----
ERROR: tuple references non-existent key
DETAIL: Trigger "check_fkeys2_pkey_exist" found tuple referencing non-existent key in "pkeys".
insert into fkeys values (10, '1', 2);
+ ERROR: could not open relation with OID 119980
insert into fkeys values (30, '3', 3);
insert into fkeys values (40, '4', 2);
insert into fkeys values (50, '5', 2);
***************
*** 87,93 ****
NOTICE: check_pkeys_fkey_cascade: 1 tuple(s) of fkeys are deleted
ERROR: "check_fkeys2_fkey_restrict": tuple is referenced in "fkeys"
update pkeys set pkey1 = 7, pkey2 = '70' where pkey1 = 10 and pkey2 = '1';
! NOTICE: check_pkeys_fkey_cascade: 1 tuple(s) of fkeys are deleted
NOTICE: check_pkeys_fkey_cascade: 1 tuple(s) of fkeys2 are deleted
DROP TABLE pkeys;
DROP TABLE fkeys;
--- 88,94 ----
NOTICE: check_pkeys_fkey_cascade: 1 tuple(s) of fkeys are deleted
ERROR: "check_fkeys2_fkey_restrict": tuple is referenced in "fkeys"
update pkeys set pkey1 = 7, pkey2 = '70' where pkey1 = 10 and pkey2 = '1';
! NOTICE: check_pkeys_fkey_cascade: 0 tuple(s) of fkeys are deleted
NOTICE: check_pkeys_fkey_cascade: 1 tuple(s) of fkeys2 are deleted
DROP TABLE pkeys;
DROP TABLE fkeys;

======================================================================

and logs show:

ERROR: 23514: new row for relation "check_tbl" violates CHECK
constraint "check_con"
LOCATION: ExecConstraints, execMain.c:1698
ERROR: 09000: tuple references non-existent key
DETAIL: Trigger "check_fkeys2_pkey_exist" found tuple referencing
non-existent key in "pkeys".
LOCATION: check_primary_key, refint.c:214
ERROR: 23514: new row for relation "check_tbl" violates CHECK
constraint "check_con"
LOCATION: ExecConstraints, execMain.c:1698

ERROR: XX000: could not open relation with OID 119980
LOCATION: relation_open, heapam.c:459

ERROR: 23502: null value for attribute "aa" violates NOT NULL
constraint
LOCATION: ExecConstraints, execMain.c:1686
ERROR: 09000: tuple references non-existent key

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> I am seeing the following parallel regression test failures. Any idea
> on the cause?

For the record, I believe this is explained by the bug I just fixed in
_bt_search().

The bug occurs only when one backend is trying to search a btree index
at the same time another backend is doing the first page split in that
index (that is, when the aboriginal root-and-leaf page gets split into
two leaf pages). In the present form of the parallel regression tests,
pg_class_oid_index and pg_type_oid_index suffer that split during the
third group of parallel tests, which is why the failures were bunched
in constraints/triggers/vacuum.

My guess is that the reason different vintages of CVS show or don't show
the problem is that modifications of the test scripts have caused more
or fewer pg_class and pg_type entries to get created, possibly moving
the critical split point before or after that set of parallel tests.
If the split occurs during a sequential test step then we'd never see
a failure. This may explain why we've not become aware of the bug till
now, even though it's certainly been there a long time.

We need to think about whether this bug is serious enough to justify a
quick 7.3.5 release. I'm leaning to the idea that it is not, because
if it were, we'd have heard about it from the field before now. In
pre-7.4 code there is only one instant in the lifespan of an index where
the bug could occur, and then only if the index is created empty.

regards, tom lane

Tom Lane wrote:
> We need to think about whether this bug is serious enough to justify a
> quick 7.3.5 release. I'm leaning to the idea that it is not, because
> if it were, we'd have heard about it from the field before now. In
> pre-7.4 code there is only one instant in the lifespan of an index where
> the bug could occur, and then only if the index is created empty.

Agreed, I don't think 7.3.5 is warranted, but it would have been nice to
get this in 7.3.4. Let's keep our eyes open for maybe a 7.3.5 later.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

"Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us> wrote:
> Tom Lane wrote:
> > We need to think about whether this bug is serious enough to justify a
> > quick 7.3.5 release. I'm leaning to the idea that it is not, because
> > if it were, we'd have heard about it from the field before now. In
> > pre-7.4 code there is only one instant in the lifespan of an index where
> > the bug could occur, and then only if the index is created empty.
>
> Agreed, I don't think 7.3.5 is warranted, but it would have been nice to
> get this in 7.3.4. Let's keep our eyes open for maybe a 7.3.5 later.

7.3.4 is already out, I agree that do a 7.3.5 build is not necessary.

Regards
Gaetano Mendola

"Mendola Gaetano" <mendola(at)bigfoot(dot)com> writes:
> "Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us> wrote:
>> Tom Lane wrote:
>>> We need to think about whether this bug is serious enough to justify a
>>> quick 7.3.5 release. I'm leaning to the idea that it is not, because
>>> if it were, we'd have heard about it from the field before now. In
>>> pre-7.4 code there is only one instant in the lifespan of an index where
>>> the bug could occur, and then only if the index is created empty.
>>
>> Agreed, I don't think 7.3.5 is warranted, but it would have been nice to
>> get this in 7.3.4. Let's keep our eyes open for maybe a 7.3.5 later.

> 7.3.4 is already out, I agree that do a 7.3.5 build is not necessary.

I dug back in the archives and convinced myself that this bug existed as
far back as Postgres 4.2 (nbtsearch.c dated 1994-02-07); it's probably
much older but I have nothing to look at to check. Since it's managed
to go undetected for a decade, it's probably not a "must fix today!"
kind of bug.

I do think we ought to put out a 7.3.5 that includes the fix at some
point, but let's wait a few weeks and see if anything else turns up
to include.

regards, tom lane

On Fri, Jul 25, 2003 at 05:47:50PM -0400, Bruce Momjian wrote:
> I am seeing the following parallel regression test failures. Any idea
> on the cause?

I think I saw about the same thing once, but I run the test again
and it didn't show up anymore at all. I'm not sure what it
exactly was, but it looked a bit simular to yours.

Kurt