There are several ways to convert bytes to trytes, caused by the fact that you can take a different number of bytes at once to convert (the more bytes you take, the less space is wasted).
In theory, you will need
log 256 / log 3 = 5.047438028571 trits for each byte.
bytes as starting point
The implementation in ascii2trytes uses 1 byte and converts it to 2 trytes (6 trits), by dividing the byte value by 27 and using the remainder as first tryte and the division result as second tryte.
Then represent these trytes in balanced ternary (0 =
000, 1 =
+--, etc.) if you really want trits :)
That way, every byte can be converted to trytes, but not every combination of trytes can be converted to bytes. One byte will use 6 trits.
This method is used to encode binary data (like small files) into a transaction message/signature.
trytes as starting point
You can also start with trytes (this method is used to send a transaction - which consists of trytes - via UDP to the neighbors). In this case, create groups of 5 trits (if you have trytes, take every group of consecutive 5 trytes and make 3 groups of 5 trits from it), then encode them to a number from 0 to 242, and put them into a byte.
That way, every tryte combination can be converted to bytes, but there are some byte results (243 to 255) that are never achieved.
In case your language supports it, you can also use bignum arithmetic to convert an arbitrary number of bytes at once to trytes and vice versa. Just multiply by 256 or 27 and add the next tryte/byte. When you are finished (and have a really big number), divide by the other value (27 or 256) and thereby split the values apart. Note that when doing so, leading zero bytes (or trytes) may get lost during roundtrip conversion, so if that is an issue, add a byte (or tryte) with value 1 at the beginning (and strip it off after finishing the convesion back).
This method is essentially the same as you use when trying to output a huge binary number in decimal. While it works, I don't know any practical need in iota applications (since either you have trytes you need to transport-encode as bytes, or bytes you need to transport-encode as trytes, and the other two methods are fine for that).