How many rounds must someone time out without making a move

before they are removed from infinity and other tournaments?

I recall once reading from Richard that it is several weeks, but based on empirical findings I believe that they are only removed if they don't make moves in any other games either. If someone has given up a particular game's infinity tournament but still play other games, they might not be removed. Perhaps some have forgotten their participation or are not adept enough with computers to know how to remove themselves from the tournament, etc.

You should only be paired with players like these in the beginning, so after a few rounds of wins they should become irrelevant.

In tournaments with few active players, they remain relevant.