Here Отца is in Genitive. "In the name of Father". The words after "of" most often are translated in Genitive.
Yes, after в/во either Prepositional or Accusative can be used. The different cases make different meanings, for example в+Prepositional can mean "in/inside something" (static position) while в+Accusative can mean "to/into something" (target of the direction). But that preposition can have a whole bunch of other meanings (as well as "in" in English). All I can advice is just remember that kind of usage: "in the name" = "во имя" (Accusative). It is not a kind of formal logic, it is a way of thinking. After you get used to it you will be able to manipulate it intuitively. Compare:
Во имя Отца = in the name of Father. (Accusative)
Что в имени моем? = What's in my name? (Prepositional)