大模型笔记4 长文本滑动窗口

Answer: {'text': ['Saint Bernadette Soubirous'], 'answer_start': [515]}

inputs = tokenizer(

    question,

    context,

    max_length=100,

    truncation="only_second",

    stride=50,

    return_overflowing_tokens=True,

)

for ids in inputs["input_ids"]:

    print(tokenizer.decode(ids))

'[CLS] To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? [SEP] Architecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend " Venite Ad Me Omnes ". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basi [SEP]'

'[CLS] To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? [SEP] the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend " Venite Ad Me Omnes ". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin [SEP]'

'[CLS] To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? [SEP] Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive ( and in a direct line that connects through 3 [SEP]'

'[CLS] To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? [SEP]. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive ( and in a direct line that connects through 3 statues and the Gold Dome ), is a simple, modern stone statue of Mary. [SEP]'

inputs = tokenizer(

    question,

    context,

    max_length=100,

    truncation="only_second",

    stride=50,

    return_overflowing_tokens=True,

    return_offsets_mapping=True,

)

inputs.keys()

dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'offset_mapping', 'overflow_to_sample_mapping'])

answers = raw_datasets["train"][2:6]["answers"]

start_positions = []

end_positions = []

for i, offset in enumerate(inputs["offset_mapping"]):

    sample_idx = inputs["overflow_to_sample_mapping"][i]

    answer = answers[sample_idx]

    start_char = answer["answer_start"][0]

    end_char = answer["answer_start"][0] + len(answer["text"][0])

    sequence_ids = inputs.sequence_ids(i)

    # Find the start and end of the context

    idx = 0

    while sequence_ids[idx] != 1:

        idx += 1

    context_start = idx

    while sequence_ids[idx] == 1:

        idx += 1

    context_end = idx - 1

    # If the answer is not fully inside the context, label is (0, 0)

    if offset[context_start][0] > start_char or offset[context_end][1] < end_char:

        start_positions.append(0)

        end_positions.append(0)

    else:

        # Otherwise it's the start and end token positions

        idx = context_start

        while idx <= context_end and offset[idx][0] <= start_char:

            idx += 1

        start_positions.append(idx - 1)

        idx = context_end

        whi