Describe the bug
Occasionally pipe overflow errors are produced when running on Linux, for example:
EVS Port1 1980-012-14:03:28.50019 66/1/CFE_SB 25: Pipe Overflow,MsgId 0x8ad,pipe TO_TLM_PIPE,sender HS
EVS Port1 1980-012-14:03:28.50019 66/1/CFE_SB 25: Pipe Overflow,MsgId 0x890,pipe TO_TLM_PIPE,sender MD
EVS Port1 1980-012-14:03:28.50019 66/1/CFE_SB 25: Pipe Overflow,MsgId 0x887,pipe TO_TLM_PIPE,sender MM
To Reproduce
Run CFS with full sample config (all apps)
Expected behavior
No pipe overflow errors
System observed on:
Debian 13
Additional context
I caught this error in the debugger just as it got the error from the system call to mq_timedsend(). At that moment, the destination queue was NOT full - in fact it was completely empty according to a debug/test tool executed separately:
43853.TO_TLM_PIPE
Maximum # of messages on queue: 10
Maximum message size: 8
# of messages currently on queue: 0
For POSIX/Linux this uses a technique of calling mq_timedsend with a timeout that has already expired (i.e. set to 0). Per the man page, the timeout should not be relevant unless the queue is full. However, at least in the version of glibc running here, it appears something else must use the timeout such that the timeout is evaluated even if the queue is not full.
When I hack the code to set this timeout 1ms in the future rather than set it to 0, I never get any pipe overflow messages.
This strongly suggests that the technique of using mq_timedsend() with an already-expired timeout does not always work the same way. There are at least some systems where this may return ETIMEDOUT even if the queue is not full.
Should investigate possibly using an alternative mechanism such as select/poll to implement blocking on message queues.
Reporter Info
Joseph Hickey, Vantage Systems, Inc.
Describe the bug
Occasionally pipe overflow errors are produced when running on Linux, for example:
To Reproduce
Run CFS with full sample config (all apps)
Expected behavior
No pipe overflow errors
System observed on:
Debian 13
Additional context
I caught this error in the debugger just as it got the error from the system call to
mq_timedsend(). At that moment, the destination queue was NOT full - in fact it was completely empty according to a debug/test tool executed separately:For POSIX/Linux this uses a technique of calling
mq_timedsendwith a timeout that has already expired (i.e. set to 0). Per the man page, the timeout should not be relevant unless the queue is full. However, at least in the version of glibc running here, it appears something else must use the timeout such that the timeout is evaluated even if the queue is not full.When I hack the code to set this timeout 1ms in the future rather than set it to 0, I never get any pipe overflow messages.
This strongly suggests that the technique of using
mq_timedsend()with an already-expired timeout does not always work the same way. There are at least some systems where this may return ETIMEDOUT even if the queue is not full.Should investigate possibly using an alternative mechanism such as select/poll to implement blocking on message queues.
Reporter Info
Joseph Hickey, Vantage Systems, Inc.