Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BigBirdPegasus #10991

Merged
Merged
Changes from 15 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
84b8ab3
init bigbird pegasus
vasudevgupta7 Mar 30, 2021
5c0307b
add debugging nb ; update config
vasudevgupta7 Apr 2, 2021
ee47797
init conversion
vasudevgupta7 Apr 2, 2021
677b72a
update conversion script
vasudevgupta7 Apr 4, 2021
94861d6
complete conversion script
vasudevgupta7 Apr 5, 2021
7529516
init forward()
vasudevgupta7 Apr 7, 2021
2ce502b
complete forward()
vasudevgupta7 Apr 10, 2021
72807a7
add tokenizer
vasudevgupta7 Apr 10, 2021
0f00e43
add some slow tests
vasudevgupta7 Apr 12, 2021
3bee313
commit current
vasudevgupta7 Apr 13, 2021
2f72512
fix copies
vasudevgupta7 Apr 13, 2021
af47e3c
add docs
vasudevgupta7 Apr 13, 2021
bbe710f
add conversion script for bigbird-roberta-summarization
vasudevgupta7 Apr 13, 2021
7b1ecce
remove TODO
vasudevgupta7 Apr 16, 2021
1de6954
small fixups
vasudevgupta7 Apr 17, 2021
fb10be0
correct tokenizer
patrickvonplaten Apr 19, 2021
c78bf6f
add bigbird core for now
vasudevgupta7 Apr 22, 2021
38c7638
fix config
vasudevgupta7 Apr 23, 2021
0b87a70
fix more
vasudevgupta7 Apr 30, 2021
c403796
revert pegasus-tokenizer back
vasudevgupta7 Apr 30, 2021
30a586a
make style
vasudevgupta7 Apr 30, 2021
9618f8b
everything working for pubmed; yayygit status
vasudevgupta7 Apr 30, 2021
53d99b5
complete tests finally
vasudevgupta7 Apr 30, 2021
a83e681
remove bigbird pegasus tok
patrickvonplaten May 4, 2021
789d170
correct tokenizer
patrickvonplaten May 4, 2021
324cb89
correct tests
patrickvonplaten May 4, 2021
24f7931
add tokenizer files
patrickvonplaten May 4, 2021
94326e1
merge conflicts
patrickvonplaten May 4, 2021
0657b81
finish make style
patrickvonplaten May 4, 2021
0aaddf4
fix test
patrickvonplaten May 4, 2021
0c11023
update
patrickvonplaten May 4, 2021
2b23619
make style
patrickvonplaten May 4, 2021
53d3a45
fix tok utils base file
patrickvonplaten May 4, 2021
2d64dda
make fix-copies
patrickvonplaten May 4, 2021
5157815
clean a bit
vasudevgupta7 May 4, 2021
cd2a5b6
small update
vasudevgupta7 May 4, 2021
237c118
fix some suggestions
vasudevgupta7 May 4, 2021
2d217e3
add to readme
vasudevgupta7 May 4, 2021
11d9c2e
fix a bit, clean tests
vasudevgupta7 May 5, 2021
b556fc9
fix more tests
vasudevgupta7 May 5, 2021
a066db7
Update src/transformers/__init__.py
patrickvonplaten May 6, 2021
9c0b312
Update src/transformers/__init__.py
patrickvonplaten May 6, 2021
4d706f7
make fix-copies
patrickvonplaten May 6, 2021
9a52751
complete attn switching, auto-padding left
vasudevgupta7 May 6, 2021
b13e76a
make style
patrickvonplaten May 6, 2021
97e1f71
fix auto-padding test
vasudevgupta7 May 6, 2021
0ba203a
make style
vasudevgupta7 May 6, 2021
0a85bb2
fix batched attention tests
patrickvonplaten May 6, 2021
89e9d5c
Merge branch 'add_bigbird_pegasus' of https://github.com/vasudevgupta…
patrickvonplaten May 6, 2021
7d2cae9
put tolerance at 1e-1 for stand-alone decoder test
patrickvonplaten May 6, 2021
34c3a63
fix docs
patrickvonplaten May 6, 2021
c229de1
fix tests
patrickvonplaten May 6, 2021
1d8d60d
correct slow tokenizer conversion
patrickvonplaten May 6, 2021
dfb74ea
Apply suggestions from code review
vasudevgupta7 May 6, 2021
ca77ad3
complete remaining suggestions
vasudevgupta7 May 6, 2021
2eb7563
fix test
patrickvonplaten May 7, 2021
File filter
Filter file types
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.

Always

Just for now

@@ -252,6 +252,8 @@ TensorFlow and/or Flax.
+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
| BigBird | ✅ | ❌ | ✅ | ❌ | ❌ |
+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
| BigBirdPegasus | ✅ | ✅ | ✅ | ❌ | ❌ |
+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
| Blenderbot | ✅ | ❌ | ✅ | ✅ | ❌ |
+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
| BlenderbotSmall | ✅ | ❌ | ✅ | ✅ | ❌ |
@@ -0,0 +1,112 @@
..
Copyright 2021 The HuggingFace Team. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
BigBirdPegasus
-----------------------------------------------------------------------------------------------------------------------

Overview
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The BigBird model was proposed in `Big Bird: Transformers for Longer Sequences <https://arxiv.org/abs/2007.14062>`__ by
Zaheer, Manzil and Guruganesh, Guru and Dubey, Kumar Avinava and Ainslie, Joshua and Alberti, Chris and Ontanon,
Santiago and Pham, Philip and Ravula, Anirudh and Wang, Qifan and Yang, Li and others. BigBird, is a sparse-attention
based transformer which extends Transformer based models, such as BERT to much longer sequences. In addition to sparse
attention, BigBird also applies global attention as well as random attention to the input sequence. Theoretically, it
has been shown that applying sparse, global, and random attention approximates full attention, while being
computationally much more efficient for longer sequences. As a consequence of the capability to handle longer context,
BigBird has shown improved performance on various long document NLP tasks, such as question answering and
summarization, compared to BERT or RoBERTa.

The abstract from the paper is the following:

*Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP.
Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence
length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that
reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and
is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our
theoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire
sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to
8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context,
BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also
propose novel applications to genomics data.*

Tips:

- For an in-detail explanation on how BigBird's attention works, see `this blog post
<https://huggingface.co/blog/big-bird>`__.
- BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using
**original_full** is advised as there is no benefit in using **block_sparse** attention.
- The code currently uses window size of 3 blocks and 2 global blocks.
- Sequence length must be divisible by block size.
- Current implementation supports only **ITC**.
- Current implementation doesn't support **num_random_blocks = 0**

The original code can be found `here <https://github.com/google-research/bigbird>`__.

BigBirdPegasusConfig
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.BigBirdPegasusConfig
:members:


BigBirdPegasusTokenizer
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.BigBirdPegasusTokenizer
:members: build_inputs_with_special_tokens, get_special_tokens_mask,
create_token_type_ids_from_sequences, save_vocabulary


BigBirdPegasusTokenizerFast
This conversation was marked as resolved by vasudevgupta7

This comment has been minimized.

@patrickvonplaten

patrickvonplaten May 4, 2021
Member Outdated

Does the fast tokenizer work?

This comment has been minimized.

@vasudevgupta7

vasudevgupta7 May 4, 2021
Author Contributor Outdated

No, will remove it.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.BigBirdPegasusTokenizerFast
:members: build_inputs_with_special_tokens, get_special_tokens_mask,
create_token_type_ids_from_sequences, save_vocabulary


BigBirdPegasusModel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.BigBirdPegasusModel
:members: forward


BigBirdPegasusForConditionalGeneration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.BigBirdPegasusForConditionalGeneration
:members: forward


BigBirdPegasusForSequenceClassification
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.BigBirdPegasusForSequenceClassification
:members: forward


BigBirdPegasusForQuestionAnswering
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.BigBirdPegasusForQuestionAnswering
:members: forward


BigBirdPegasusForCausalLM
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.BigBirdPegasusForCausalLM
:members: forward


ProTip! Use n and p to navigate between commits in a pull request.