Homeworld can run in windowed mode using the -window option:
- Right click on your Homeworld shortcut and select properties
- In the target field add a space then add:
(example: C:\Program Files\Sierra\Homeworld\homeworld.exe -window).
- Click apply an start the game from the shortcut.
If you have an issues with running Homeworld in windowed mode, please post them in the comments below. The more information the better (especially the target field of your shortcut)!