Approaching under the pretext of a bus stop in a minibus

Today, the man sitting next to me on the minibus started asking about the stops continuously. But every time he asked, it felt like he was touching my arm or shoulder. At first, I pretended not to notice, but then I got tired of it and switched seats. Friends, do these things always happen ‘accidentally’ or do they make it a habit to annoy people? Do you think I should say something in such situations, or is just changing seats enough?