These days I attend matches under the guise (in my head), of a scout. When I think I've seen enough, I'm off. Not waiting to leave with the masses.
I went to watch Rotherham v Southampton in the League Cup when they played at Don Valley, just to see what it was like watching a match at that stadium. (rubbish, by the way), I left after twenty mins. Job done. Didn't upset anyone, no traffic.
If it bothers people, like it does my mate, I tell him to convince himself they are just going to the toilet, because that's allowed apparently. It seems to ease his mild anger and frustration.
There, so just think, 'Ah, toilet visit', and you'll be fine.