The 905-1180 Å spectral range of FUSE includes numerous transitions of molecular hydrogen, making it possible to study H2 in diffuse interstellar environments directly through absorption measurements. We have searched for H2 absorption in five starburst galaxies: NGC 1705, NGC 3310, NGC 4214, M83 (NGC 5236), and NGC 5253. We tentatively detect weak absorption by H2 in M83 and NGC 5253 and set upper limits on the H2 column density in the other galaxies. Conservative upper limits on the mass of molecular gas detected with FUSE are many orders of magnitude lower than those of the H2 mass inferred from CO emission measurements for the four galaxies in our sample in which CO has been detected. This indicates that almost all the H2 is in the form of clouds with N(H2)≳ 1020 cm-2 that are opaque to far-UV light and therefore cannot be probed with far-UV absorption measurements. The far-UV continuum visible in the FUSE spectra passes between the dense clouds, which have a covering factor less than 1. The complex observational biases related to varying extinction across the extended UV emission in the FUSE apertures prevent an unambiguous characterization of the diffuse H 2 in these starbursts. However, the evidence suggests that there is less H2 in the diffuse interstellar medium between the dense clouds compared with similarly reddened sight lines in the Milky Way. This holds with the expectation that the destructive UV radiation field is stronger in starbursts. However, previous UV observations of these starbursts have shown that there is reddening caused by the diffuse interstellar medium. This suggests that, while diffuse H2 may be destroyed in the starburst, dust still exists.