Not all Social Security recipients have to pay taxes on their benefits, but if your income is above certain limits, then a portion of what you get in Social Security is subject to tax. To figure out what you owe, take half your Social Security benefits and add in your other income. For single filers, if the result is between $25,000 and $34,000, you might have to add in up to half your benefits as taxable income. Above $34,000, as much as 85 percent of your benefits could be taxable. The corresponding figures for joint filers are $32,000 and $44,000.