A cover letter should be short, simple, and to the point. Generally, it should state your business and briefly introduce who you are as a writer. Always read the journal's writers' guidelines to see if they have any specific requirements for the cover letter.
As with any letter, address it to a specific person if possible. Use the name of the editor appropriate for your work. You can find this information on the masthead of the journal. Some of the university-affiliated journals have an editorship that changes yearly, so it’s not always clear who will be in charge when your submission arrives. If you can't find the appropriate name or if you're not sure, you can address the letter to the more general title of the editor of the genre, such as Fiction Editor or Poetry Editor. (Some journals prefer this. Remember, always read and follow the writers' guidelines.)
Keep the body of the letter short. Indicate that you are submitting a short story (or poem) for their consideration. Don't explain what the story or poem is about. Let the work stand on its own. A mention of why this work is particularly relevant for the specific literary journal is also helpful. It demonstrates you know the publication and have put thought into where to submit.
Next, give a brief run down of your qualifications as a writer. If you have published before, mention where your work has appeared. If you have several publications, you don't need to list all of them. Just select a few. Also, include any awards or honors you've received for your writing. If you don't have a lot of accolades yet, don't worry. Focus on whatever experience you do have—like taking creative writing classes—and don't apologize for or otherwise point out your inexperience.
End by thanking the editor for his or her consideration. Keep it professional and friendly.