I think that body language has a special place in genre fiction, where we're so often describing creatures and species--that the reader hasn't ever seen before. Genres such as fantasy, science fiction and horror add a new dimension to body language in writing. Because it is such an important factor in human communication, one has to wonder how it would function in the interaction between species.
Even other species--aliens, mythical species, etc.--that are similarly shaped could have a completely different take on body language, gesture and facial expression. Other species with different body-forms, different methods of communication, and different limbs probably don't view it the same way we do.
Take cats for instance, as with many predators their vision is adapted to look for movement. This makes them particularly adept at reading visual body language cues. They communicate with flicks of the tail, eye movements and changes in posture that can be very subtle. For the most part, they only meow to communicate with us. Although, they have a full range of other vocal communications, some of which are too high for us to hear. If I were to create a species based on cats, they would be heavily dependent on body language and able to communicate well through gestures or movements. (And smells, but that's a totally different post!)
Even species that aren't meeting for the first time have a huge chance of miscommunication, lack of understanding, etc. Especially if the characters aren't trained in communicating with other species.
Of course, introducing body language is no different than introducing a language, and if it's so different that the reader can't understand it, you're going to have a tough time. Introducing this new form of language could be tricky. You'll need to introduce specific gestures and repeat them in a consistent and visible context. Possibly even having a POV character observe it. Doing this, however, can help to characterize the species you're introducing.
For instance, if your characters communicate telepathically among themselves and do not need to see one another to do it, they may have very little body language at all. What would that look like to a human who is expecting, even depending on, those cues to understand what's being communicated? Would they find it disturbing to speak to someone and not get those cues? It must be a little like communicating over the internet, and we all know how difficult that can sometimes be.
Body language, gesture and expression should spring naturally from the other methods of communication used by the species. And, in turn, those methods of communication should correspond to the species dominant senses. Unless, of course, communication is not vital to the species, which in itself is an interesting trait and certainly worth exploring.
Consider the methods of communication your species use, especially if you'll be using their viewpoint. Many species have a dominant means of communication, and supplimental ones. Species that can't make noise will have trouble communication over long distances, unless they're telepathic. Species that communicate by scent might be able to communicate at long distance, but how do they deal with communicating face to face?
And how would that communication hinder or aid their dialogues with the other species in the story? These details can sometimes seem to get in the way of your plot, but if you use them and work them into the story, they can give your created species and the characters within that species, added depth and definition.
Body language can also come into play in fantasy, science fiction and horror when animals are used as characters. Because animals use their senses differently than we, their methods of communication are different as well. Dogs communicate by scent and sound, so much so that humans listening to pre-recorded dog sounds can often distinguish the emotional context in which the sound was made. On the other hand, dogs carefully watch our body language and have no trouble learning gestured commands without a verbal cue.
Different animals respond to different cues and respond better to different forms of body language and non-verbal communication.
How do you use body language when dealing with non-human species? Do you find yourself using similar methods or do you try different approaches?