There has been only one time I've ever rented a car on my own. My husband handles all our rentals when we travel but I'm pretty observant and have picked up a lot of the tips he has to offer.
First, shop around. Rates vary a lot as you can imagine. Be sure to compare apples to apples or in this case mid-size to mid-size, etc. If you are traveling out of the country be sure to check on the restrictions for where you take the car. For instance many US rental places frown on you taking a car across the border to Mexico. Overseas, you want to check the restrictions as well (some countries require a special license too). Our road trip in New Zealand included both North and South islands. We found a rental company that not only let us take the car on the ferry but included the purchase of our tickets as well. It was one less thing for us to do. And don't forget those AAA, AARP, etc. discounts!
At the counter you will be offered all sorts of options. Don't prepay for gas unless you know you're not going to be able to stop on the way back to return the car. On your way out of the rental place look around for a gas station nearby. There's always one. Mark it on your map. Even though the price may be a little higher than other stations, chances are good it will still be lower than what the rental company charges.
Insurance? Only if you really must. Check with your insurance company. It probably covers you for rentals. The credit card you use can cover things your policy doesn't and often covers you when you are abroad where your regular car insurance may not. While traveling in Australia a large bus passed us kicking up a stone that shattered the rear passenger window. Our credit card company covered all the damage.
"Want an upgrade?" If you are asked that, be sure the price is right before you agree. 'Nuf said.
Inspect your rental thoroughly before you drive out. We've taken to documenting the condition with iPhone pictures especially if the car already had some scratches or dents. If you are doing a drop off where no one inspects when you turn the car in, it might be a good idea to photograph it then too.
Those are the major things I can think of. We've never really had too many problems with rental cars. Oh, maybe just one more. Before you leave the car, check that glove compartment for anything you may have left behind in there. We lost a camera that way once.