I keep it simple and low tech. I have one Excel file with a different spreadsheet for each story. Every time I submit a story, I pull up the file, find the right spreadsheet and list the pertinent information. You can use a similar approach with whatever program you know how to use well. Some writers like to eliminate the technology entirely. I've seen one system that uses note cards and another with a very large desk calendar. Choose a method that will be easy for you to update and one that you know you will use.
For every submission, note the story, the journal and the date of submission. You might also find it helpful to include the journal's address (physical or web, depending on how you submitted), the editor to whom you directed the submission and the wait time stated in the journal's guidelines. Leave space for special notes, too. You might want to remind yourself, for instance, that a submission is a follow up to a request to see more work. As responses come in, make sure to return and update the status.
I know many writers who would argue that using a submission tracking program is more efficient and fun. They might be right. There are plenty of options out there and a quick Internet search will give you a look at the variety of approaches to tracking. Duotrope's Digest, for example, has a free submission tracker that allows you to sort your submissions by status, date sent, market and more. The submission tracker also taps into Duotrope's data on the average response time for a given market based on user's reports and the maximum response time estimated by editors at the journal.
Whatever system you use, you want to make it easy to see where your story is under consideration. Should a story be accepted, you know exactly which journals are considering that story and can quickly send notes withdrawing the submission. Many writers find keeping track of submissions also helps them stay on top of the process. You can see—with just a glance—which stories are making the rounds and which need a boost back out.