Do I have to connect this piece on the exhaust to something Or can I plug it somehow? The car has no emissions on it. There's also one on the opposite side too.

 

That is the AIR(air injection reaction) inlet to your exhaust manifold. Whether or not you connect it depends on whther or not you need to pass emmissions inspection. Do you still have an AIR pump? If you dont need to pass emmissions and the air pump is gone, remove the tubing and plug the holes in the cast manifold with 1/4" pipe plugs. You can leave it as is but if the check valve fails, it will be an exhaust leak.

 

You can also leave the small tubes in the manifolds, and unscrew that piece (the check valve), and get a standard metal pipe cap from plumbing section at Home Depot etc. that will cap it off - not as pretty as taking the tubes off the manifolds, but only need 1 cap per side instead of 4 plugs per side.