Imagine that you’ve just encountered an alien life form. Once you’ve gotten over the initial shock/excitement/etc. of discovering that we are indeed not alone in the universe, you of course want to find out all about it: where it comes from, what its home world is like. Assuming it’s sentient, you’ll also want to know what kind of society it comes from, what kind of life its people lead, and what kind of technology they have available. Clearly the best way to find out all this information is to ask. But first you have to learn the language.
In fictional universes the languages are often in many ways just like English or German or Finnish, and in many cases they even are ours, just slightly modified (e.g., Galactic Basic in Star Wars, High Speech in The Dark Tower series). However, there are plenty of earthly languages more different from English and Finnish than are science fiction and fantasy languages such as Mandalorian and Quenya. Authors typically choose anthropomorphic protagonists and antagonists because it’s more difficult to empathize with and understand the motivations of characters who are too wholly other. But contrary to the route many authors choose, the steps involved in communicating with an alien being would not simply be “learn the language”.
First you’d have to discover if they even had a language. This might not be as simple as it sounds, because they might communicate in a way that humans could not perceive as structured communication (telepathy, chemical signals like those many insects use, electromagnetic emission above or below the visible light spectrum – below 380nm or above 760nm). Humans tend to make a lot of assumptions about the universality of certain features of languages (yes – even linguists), and many of these assumptions are not well-founded even for human languages, much less necessarily valid for alien languages.
Even in the simple case of a lifeform emitting some type of sound, you’d have to figure out whether or not it was communication, and then whether or not it was language. Noise emission could be unintentional, as humans radiate heat, or it could be a response to the environment or the lifeform’s internal state. And if it does have a repeating pattern of some sort, the question would still remain as to whether or not it was limited to a finite series of fixed calls, as in monkeys and dogs, or whether it was an infinitely variable system of communication, as in human language.
This is an important point – while most authors choose the assume that a sentient race would have developed language akin to human language, this is not necessarily a fixed universal. It would certainly possible for a moderately intelligent and sentient race of beings to not have developed a complex language like those humans use. Linguists and neuroscientists have firmly established that there is a genetic component to language, and that the ability to acquire and use language is not strictly correlated with intelligence. This is why of all the species on Earth only humans have an infinitely variable arbitrary system of communication that can be used to refer to distal points in space and time.
Even once language the existence of a language per se has been established, there’s still the question of perceiving and reproducing the sounds in that language. An alien lifeform would doubtless have an acoustic production tract significantly different from humans, and thus it’s not a given that humans would be even physically able to distinguish the sounds it produced, much less reproduce them. Certainly there would be ways around this. If the language is spoken in the 40k-80k Hz range, you could simply pitch shift a recording down to the human hearing range. More difficult might be a case in which the language was spoken in a very narrow range, say 300-350 Hz, with 10 or 15 tonal distinctions within that range. On the other hand, we might not expect any such language to exist, regardless of differences in cognitive processing and acoustic production, since the natural evolution of any system of communication should favor ease of perceptibility.
In addition, there’s no reason to think that the facts on phonology (pronunciation and patterning of sounds in a language), syntax (the order of words in a language), or semantics (meaning) would be anything like human language. While many linguists believe in a Universal Grammar underlying all human languages, alien languages would not necessarily have the same Universal Grammar, since they would have evolved a different Language Acquisition Device that could have different grammar rules in it, rules that might make no sense from a human language standpoint, or even from a human cognition standpoint.
What does this mean for the author? Don’t limit yourself to what you know – or better yet, expand your knowledge beyond what’s close at hand. While English and other European languages have certain basic features, such as fixed word order within a sentence, tense marking on verbs that shows when in time an action happened, and sounds that use the lips, not all human languages have these features. Some languages allow the subject, verb, and object to appear in any order. Some languages have no fixed way of indicating when in time an event occurred, and may focus instead on the state of completion of the event itself, rather than its temporal relation to the speech time. Some languages have no sounds that use the lips, e.g., p, b, f, v, and some languages have no nasal sounds, e.g., m, n. Given that some human languages, like Tillamook, Blackfoot, and Warlpiri, seem so alien to speakers of English, Spanish, or Hindi, it’s good to keep in mind that alien languages would probably even more, well, alien.