Sale by Private Treaty means that you set a price at which your property is to be marketed to the public through the marketing plan agreed to by you and the agent.
If you submit your property for Auction this means that prospective purchasers will bid against one another at a date and time suitable to you. You can set a reserve price, which is the minimum you will accept, and once bidding has passed that level you know you have a sale, as it will be unconditional and a predetermined deposit must be paid on the day.