Do I have to pay for video conversions that resulted in an error?

This depends on the type of error that has occurred.  If the error reported is during the upload process or because you are attempting to upload an unsupported format, there is no charge.  Otherwise, you will only be charged for the source video encoding bandwidth and not for the output files that have resulted in error.