A genre is a category of literature, such as mystery, suspense, science fiction or horror. Each genre has its own conventions. Romance, for example, focuses on romantic love between two people and often ends positively. Generally, genre fiction tends to place value on entertainment and, as a result, it tends to be more popular with mass audiences.
Literary fiction, on the other hand, is a bit trickier to define. In general, it emphasizes meaning over entertainment. Literary fiction also aspires toward art. Of course, that abstract of “art” is where things get most tricky. What is art? In fiction it can be defined as interesting and deep manifestations of the elements of craft: dimensional characters, a pleasing arc of tension, evocative language and thematic purpose.
Of course, literary and genre fiction aren’t exclusive of one another. A work of genre fiction can be literary as well. Jane Austen, for example, wrote literary romances, like Pride and Prejudice.
This issue becomes most important when you begin submitting your work. Some publishers embrace certain genres, while others don’t. Look at books that have many qualities similar to your own work and see how those are classified.