Optical fibers or integrated optical waveguides have arbitrary cross-sectional index or refraction distribution. An efficient finite element method for analyzing the propagation characteristics of dielectric / optical waveguides with open boundary is presented. The propagation modes are hybrid, for which a variational expression is formulated in terms of the longitudinal electric and magnetic field components. Infinite elements are introduced to consider open boundary or to extend the region to infinity. Several specific examples are given and the results are compared with those obtained by other approximate methods. Very close agreements have been found.