Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BigBirdPegasus #10991

Merged
Merged

Conversation

@vasudevgupta7
Copy link
Contributor

@vasudevgupta7 vasudevgupta7 commented Mar 31, 2021

What does this PR do?

This PR will add Google's BigBird-Pegasus. Extending #10183

Following checkpoints will be added:

It is verified that uploaded models work correctly, see:

Here a notebook showing how well BigBirdPegasus works on long-document summarization: https://colab.research.google.com/github/vasudevgupta7/bigbird/blob/main/notebooks/bigbird_pegasus_evaluation.ipynb

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline, Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten

@vasudevgupta7 vasudevgupta7 mentioned this pull request Apr 7, 2021
5 tasks done
@vasudevgupta7
Copy link
Contributor Author

@vasudevgupta7 vasudevgupta7 commented Apr 10, 2021

For running conversion script for BigBirdPegasus:

python3 src/transformers/models/bigbird_pegasus/convert_bigbird_pegasus_tf_to_pytorch.py --tf_ckpt_path src/tf_ckpt/bigbird-pegasus-large-arxiv/model.ckpt-300000 --save_dir src/google/bigbird-pegasus-large-arxiv

For running conversion script for bigbird-roberta EncoderDecoderModel:

python3 src/transformers/models/bigbird_pegasus/convert_bigbird_roberta_tf_to_pytorch.py --tf_ckpt_path src/tf_ckpt/bigbird-roberta-arxiv/model.ckpt-300000 --save_dir src/google/bigbird-roberta-arxiv
@lefnire lefnire mentioned this pull request Apr 20, 2021
3 tasks
@patrickvonplaten patrickvonplaten changed the title [WIP] Add BigBirdPegasus Add BigBirdPegasus May 6, 2021
@patrickvonplaten patrickvonplaten requested review from sgugger and LysandreJik May 6, 2021
Copy link
Member

@patrickvonplaten patrickvonplaten left a comment

Amazing contribution @vasudevgupta7 !

@patrickvonplaten patrickvonplaten requested a review from patil-suraj May 6, 2021
Copy link
Member

@patil-suraj patil-suraj left a comment

Amazing work @vasudevgupta7 !

Just left a few nits.

@sgugger
sgugger approved these changes May 6, 2021
Copy link
Member

@sgugger sgugger left a comment

Great work! Thanks a lot for this addition. I left a few nits.

Copy link
Member

@LysandreJik LysandreJik left a comment

Great job @vasudevgupta7! Really cool to have BigBirdPegasus in the library as well.

Nice notebook too. Are you planning a notebook like last time as well?

@vasudevgupta7
Copy link
Contributor Author

@vasudevgupta7 vasudevgupta7 commented May 6, 2021

@LysandreJik, yes we are planning to add this notebook with a few modifications.

@vasudevgupta7
Copy link
Contributor Author

@vasudevgupta7 vasudevgupta7 commented May 7, 2021

@patrickvonplaten, Test failing on CircleCi: tests/test_modeling_bigbird_pegasus.py::BigBirdPegasusStandaloneDecoderModelTest::test_decoder_model_attn_mask_past is passing for me locally.

Everything else is fixed!!

@patrickvonplaten patrickvonplaten merged commit dc3f675 into huggingface:master May 7, 2021
14 checks passed
14 checks passed
@github-actions
run_tests_templates
Details
ci/circleci: build_doc Your tests passed on CircleCI!
Details
ci/circleci: check_code_quality Your tests passed on CircleCI!
Details
ci/circleci: check_repository_consistency Your tests passed on CircleCI!
Details
ci/circleci: run_examples_torch Your tests passed on CircleCI!
Details
ci/circleci: run_tests_custom_tokenizers Your tests passed on CircleCI!
Details
ci/circleci: run_tests_flax Your tests passed on CircleCI!
Details
ci/circleci: run_tests_hub Your tests passed on CircleCI!
Details
ci/circleci: run_tests_pipelines_tf Your tests passed on CircleCI!
Details
ci/circleci: run_tests_pipelines_torch Your tests passed on CircleCI!
Details
ci/circleci: run_tests_tf Your tests passed on CircleCI!
Details
ci/circleci: run_tests_torch Your tests passed on CircleCI!
Details
ci/circleci: run_tests_torch_and_flax Your tests passed on CircleCI!
Details
ci/circleci: run_tests_torch_and_tf Your tests passed on CircleCI!
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants