Your answer suggests that neither depression nor anxiety is a major factor for your memory loss.
Drinking too much another is another major cause of memory problems. Alcohol can cause both short-term and long-term memory problems. Current recommendations for alcohol consumption based on heart health, liver health, and breast cancer risk are no more than one drink per day for women and no more than two drinks per day for men. (One drink is one glass of wine or one bottle of beer or one shot glass of liquor.)
Do you think you might be drinking too much?