Note to self:
The Page.RegisterStartupScript just outputs the text you pass to the method and doesn't add script tags. If you don't add script tags your text will be output at the bottom of every page (which is embarrassing). Why can't I remember this....
P.S. - In case you were wondering, I just discovered this is a bug in the RsWebParts version 1.1 which will be fixed in 1.2.
P.S.S. - I hear this is “fixed” in ASP.Net 2.0...