My understanding is that "faster" simply refers to an all-new on-screen experience which includes one-stop search, but the actual processer is not any faster than the XS. Also the same "faster" interface will be available via an update in May 2013 for the XS and other Roku models. So no difference there.
As far as the remote, the Roku 3 has a headphone jack in the remote. For some that may be awesome. Personally, I would rarely, if ever, use it.
This is incorrect. The processor is much faster on the Roku 3 (Dual Core A9) than on the Roku 2 (ARM 11). I found the Hulu Channel, which is notoriously sluggish and unresponsive quite often on the Roku 2, to be very quick and responsive on the Roku 3. Of course, navigation is also very smooth on the Roku 3 compared to the Roku 2, but that alone wouldn't be worth upgrading to Roku 3 because most other channels work very well on the Roku 2 (Netflix and Amazon are examples of well designed channels.)
Personally, I think if you're a Hulu Plus subscriber or love to play Angry Birds, this is probably worth the upgrade.