This turned out to be very easy to fix: tape a piece of tinfoil to the back of my access card:

I feel a bit silly that after spending months pulling the card out each time the fix ended up taking me a couple minutes. I often decide to put up with minor annoyances instead of thinking about whether there's a way to fix them, and I think overall that has made my life substantially better, but in this case even a little thought would have been well worth it!