When you start a Duel, you are the Owner of all the cards in your Extra Deck, Main Deck, Side Deck, and during the Duel, the cards in your Graveyard. While your opponent may briefly become the Controller of one of your cards, he/she can never become the owner.

An Ante Duel, forbidden by Ethics and Tournament Policy alike, is only exception for a card to change owner during (after) a duel.