Store message definition in bag

[jhurliman]

Description

Neither the metadata.yaml file or .db3 file(s) contain the ROS2 message definitions used to generate the bag. This makes it impossible to deserialize rosbag2 files without having the correct version of all the original .msg files on the machine that is decoding the bag. Tools like Foxglove Studio are not able to decode rosbag2 files, and if message definitions are ever modified you have to store a full snapshot of message definitions alongside each bag file. If the wrong message definition is used during deserialization, there doesn't appear to be any mechanism to catch this leading to either decoding failures or potentially invalid decoding.

This is a regression from the rosbag1 format, where a .bag file is self describing and can be decoded even if the collection of .msg files used to generate it have been lost, or if a .bag file is shared with someone else.

[emersonknapp]

Thanks for filing this ticket - it's something that has come up as part of other discussions but was not explicitly tracked as its own feature request. I've re-worded the title to name the positive feature to build (rather than the negative lack of feature)

[danthony06]

@jhurliman I have not personally used this package, and there's definitely some major assumptions made in the codebase, but I think this is doing a lot of what you're envisioning:
https://gitlab.com/ternaris/rosbags

[danthony06]

Ah, looking at how they do that, it looks like they're manually embedding a bunch of information about the message format, so it can't handle arbitrary messages. I do like the idea of having all of the information to fully extra the messages from a bag embedded in the actual bag file.

[chaoflow]

@danthony06 Writing as one of the authors: rosbags so far does not store message definitions. It supports reading rosbag2 with custom message types by registering the idl/msg files, see https://ternaris.gitlab.io/rosbags/topics/typesys.html

There are two cases:

1. conversion from rosbag1 to rosbag2; message definitions from rosbag1 are used to convert and we want to store these along with topic latching info for later deserialization and to convert back to rosbag1.
2. recorded rosbag2; either the recorder or a post-processing tool needs to add the message definitions, depending on when these are available.

@emersonknapp et al: What are your thoughts on where and how to store message definitions and topic latching info?

[pkess]

Hi all,
i would like to get this feature as well. But as a first step it might be very usefull if the rosbag would store at least some kind of a hash of the message definition for each topic. With this you could check if you have the correct message definition available. Ich think rosbag1 stored both the full message definiton and some kind of hash for the message type.

[danthony06]

@chaoflow thank you for the correction, sorry I missed the details on how you're getting the format.

@pkess You're right that rosbag1 is storing the MD5 sum, but it's done on a connection basis, not message type basis, according to the documentation: http://wiki.ros.org/Bags/Format/2.0. I'm guessing this is to catch a corner case in ROS 1 where it might be possible to have nodes using the same message type, but are different versions with different internal fields. For example, Node A and B are compiled with one message definition and use that to communicate, while Nodes C and D have a message with the same name, but has been changed and built at a later time, and they can use that to communicate. So the bag file would have multiple entries for the same message name, but the implementation is different, so it has to track the MD5 of the connection.

[emersonknapp]

What are your thoughts on where and how to store message definitions and topic latching info?

I see the obvious place as being in the TopicMetadata - this thing already exists and we should be able to safely expand it with some serialized representation of the message definition.

For latching info - this is handled using the Durability QoS using TRANSIENT_LOCAL durability with a history. This behavior provides a superset of the ROS 1 "latching" functionality. http://docs.ros.org/en/galactic/Concepts/About-Quality-of-Service-Settings.html#qos-policies for some info

[durko]

@emersonknapp Saving message definitions per topic could lead to some content duplication for topics sharing message types. Nested types (e.g. std_msgs/msg/Header) are often not part of the definition itself and need to be also stored somewhere. While yaml supports anchors and references, it would be easier to work with a flat storage for message definition metadata, e.g. with a type_definitions array as a sibling to topics_with_message_count.

Suggested content:

rosbag2_bagfile_information:
  type_definitions:
  - format: 'idl'
    definition: '{content of Bool.idl file}'
  - format: 'msg'
    definition: 'MSG: std_msgs/Int32\n{content of Int32.msg file or message_definition from ROS1 bag}'

The idl case is straightforward, as only the message definition content is necessary. The msg case requires one additional piece of information, namely the message type name. In the example above the message type name is prepended to the message definition, similar to concatenated definitions in message_definition fields of ROS1 bag connection headers. Alternatively, message type names could also be saved as a separate key. As both formats can define multiple types in a single message definition, a type name key would probably have to be a list of strings.

[jhurliman]

My understanding is that DDS (and .idl files as an extension of that) is one possible transport system in ROS2 which has a pluggable transport layer. If that’s correct, making idl storage a first class concept in rosbag2 would be a leaky abstraction.

ROS1 bags handle this issue gracefully. The one thing they add beyond the raw msgdef is a checksum, so you don’t have to do a byte for byte long string comparison to see if a message definition has changed between two bags. ROS1 bags also store msgdefs per connection instead of per topic, which would fix the replay bug when two publishers publish to the same topic with different QoS profiles.

[chaoflow]

My understanding is that DDS (and .idl files as an extension of that) is one possible transport system in ROS2 which has a pluggable transport layer. If that’s correct, making idl storage a first class concept in rosbag2 would be a leaky abstraction.

@jhurliman The proposal above is not bound to idl but already contains msg as other format.

ROS1 bags handle this issue gracefully. The one thing they add beyond the raw msgdef is a checksum, so you don’t have to do a byte for byte long string comparison to see if a message definition has changed between two bags.

When considering a checksum for message definitions for rosbag2 we should take normalization into account instead of simply hashing the string definition.

ROS1 bags also store msgdefs per connection instead of per topic, which would fix the replay bug when two publishers publish to the same topic with different QoS profiles.

QoS profile is a concept orthogonal to messages definitions, same as latching in rosbag1, i.e. a topic has a message type with a specific message definition and a topic can have a QoS profile.

ROS1 stores message definitions per connection and there can be multiple connections per topic with theoretically, regarding the storage format, different message types and definitions. I have not seen mixed-type-topics in practice, am not sure whether it is possible to generate them through recording, in contrast to manually writing bags, and don't see a use case for them so far.

[danthony06]

I agree that the mixed-type-topics is an unusual case, and unlikely to happen in practice. About the only time I can think of when I have encountered this is when there are multiple robots that are running different versions of software. It's generally not a great idea and usually indicates something wrong with the larger system, but it's certainly possible.