Charcoal, 39 bytes

≔Ｅ⁺θ× ¬﹪Ｌθ³⌕βιθ⭆⪪Ｅ⁺÷θ⁹⁺÷θ³θ﹪鳦³§⁺β ↨³ι

Try it online! Link is to verbose version of code. Explanation:

≔               Assign

   θ            Input string

  ⁺             Concatenated with

                Literal space

    ×           Repeated

         θ      Input string

        Ｌ       Length

       ﹪        Modulo

          ³     Literal 3

      ¬         Logical not

 Ｅ              Mapped over characters

             ι  Current character

           ⌕    Position found in

            β   Lowercase alphabet

              θ To variable



     θ                      List of positions

    ÷                       Vectorised integer divide by

      ⁹                     Literal 9

   ⁺                        Concatenated with

         θ                  List of positions

        ÷                   Vectorised integer divide by

          ³                 Literal 3

       ⁺                    Concatenated with

           θ                List of positions

  Ｅ                         Map over values

             ι              Current value

            ﹪               Modulo

              ³             Literal 3

 ⪪                          Split into

                ³           Groups of 3

⭆                           Map over groups and join

                   β        Lowercase alphabet

                  ⁺         Concatenated with

                            Literal space

                 §          Cyclically indexed by

                       ι    Current group

                     ↨      Converted from

                      ³     Base 3

                            Implicitly print

Python 2, 180 176 174 165 163 bytes

lambda s:''.join(chr(32+(33+a*9+3*b+c)%59)for a,b,c in zip(*[iter(sum(zip(*[(c/9,c/3%3,c%3)for c in map(o,s+' '[len(s)%3:])]),()))]*3))

o=lambda c:(ord(c)%32-1)%27

Try it online!

Input can be upper or lower. Output is uppercase

Pyth, 34 33 bytes

m@+G;id3csCmtj+27x+G;d3+W!%lz3zd3

Full program. Input is expected as lowercase, output is a character array. Try it online here, or verify all test cases at once here.

m@+G;id3csCmtj+27x+G;d3+W!%lz3zd3   Implicit: z=input(), d=" ", G=lowercase alphabet

                           lz       Length of z

                          %  3      The above, mod 3

                        W!          If the above != 3...

                       +      zd    ... append a space to z

           m                        Map the elements of the above, as d, using:

                  +G;                 Append a space to the lowercase alphabet

                 x   d                Find the 0-based index of d in the above

              +27                     Add 27 to the above

             j        3               Convert to base 3

            t                         Discard first element (undoes the +27, ensures result is 3 digits long)

          C                         Transpose the result of the map

         s                          Flatten

        c                       3   Split into chunks of length 3

m                                   Map the elements of the above, as d, using:

     id3                              Convert to decimal from base 3

 @+G;                                 Index the above number into the alphabet + space

                                    Implicit print

Alternative 34 byte solution: sm@+G;id3csCm.[03jx+G;d3+W!%lz3zd3 - rather than +27 and tail, uses .[03 to pad with 0 to length 3. Can be 33 if the leading s is dropped.

Edit: saved a byte by dropping leading s as character arrays are valid output

Jelly, 29 26 25 bytes

⁶Øa;3ṗ¤,©Ṛy;⁶$L3ḍƊ¡ZFs3®y

Try it online!

How it works

⁶Øa;3ṗ¤,©Ṛy;⁶$L3ḍƊ¡ZFs3®y  Main link. Argument: s (string)



⁶                          Set the return value to space.

 Øa;                       Append it to "a...z".

    3ṗ¤                    Yield the third Cartesian power of [1, 2, 3].

       ,©                  Pair the results and store the pair in the register.

                           The register now holds

                           [[[1, 1, 1], ..., [3, 3, 3]], ['a', ... ,'z', ' '].

         Ṛ                 Reverse the outer array.

           ;⁶$             Append a space to s...

              L3ḍƊ¡        if the length is divisible by 3.

          y                Transliterate according to the mapping to the left.

                   ZFs3    Zip/transpose, flatten, split into chunks of length 3.

                       ®y  Transliterate according to the mapping in the register.

Ruby, 153 145 138 131 bytes

->a{a<<" "if a.size%3<1;a.map{|c|[(b=(c.ord%32-1)%27)/9,b%9/3,b%3]}.transpose.join.scan(/.{3}/).map{|x|((x.to_i(3)+65)%91+32).chr}}

Try it online!

A quick and naive approach, works with lowercase text. Inputs and outputs arrays of characters.

APL+WIN, 102 bytes

⎕av[n[c⍳(⊂[2]((⍴t),3)⍴,⍉⊃(c←⊂[2]c,(,⍉9 3⍴c←9/⍳3),[1.1]27⍴⍳3)[(⎕av[n←(97+⍳26),33])⍳t←t,(3|⍴t←⎕)↓' '])]]

Explanation:

t←t,(3|⍴t←⎕)↓' ' Prompts for input and applies coprime condition



(⎕av[n←(97+⍳26),33]⍳ Indices of characters in APL atomic vector 



c←⊂[2]c,(,⍉9 3⍴c←9/⍳3),[1.1]27⍴⍳3) Create a matrix of table, row column for 27 characters



⊂[2]((⍴t),3)⍴,⍉⊃ Extract columns of c corresponding to input and re-order



c⍳ Identify Column indices of re-ordered columns



⎕av[.....] Use indices back in atomic vector to give enciphered text  

Example of screen shot of test case:

⎕av[n[c⍳(⊂[2]((⍴t),3)⍴,⍉⊃(c←⊂[2]c,(,⍉9 3⍴c←9/⍳3),[1.1]27⍴⍳3)[(⎕av[n←(97+⍳26),33])⍳t←t,(3|⍴t←⎕)↓' '])]]

⎕:

'output'

rrvgivx  

Java (JDK), 192 bytes

s->{String T="",R=T,C=T,r=T;for(int c:s){c-=c<33?6:97;T+=c/9;R+=c%9/3;C+=c%3;}for(var S:(s.length%3<1?T+2+R+2+C+2:T+R+C).split("(?<=\G...)"))r+=(char)((Byte.valueOf(S,3)+65)%91+32);return r;}

Try it online!

Very naive approach. Takes a lowercase char as input but outputs a String.

Explanations

s->{                                       // char-accepting lambda

 String T="",                              //  declare variables Table as an empty string,

        R=T,                               //                    Row as an empty string,

        C=T,                               //                    Column as an empty string,

        r=T;                               //                    result as an empty string.

 for(int c:s){                             //  for each character

  c-=c<33?6:97;                            //   map each letter to a number from 0 to 25, space to 26.

  T+=c/9;                                  //   append the table-value to Table

  R+=c%9/3;                                //   append the row-value to Row

  C+=c%3;                                  //   append the column-value to Column

 }                                         //

 for(var S:                                //  For each token of...

     (s.length%3<1?T+2+R+2+C+2:T+R+C)      //    a single string out of table, row and column and take the space into account if the length is not coprime to 3...

      .split("(?<=\G...)"))               //    split every 3 characters

  r+=(char)((Byte.valueOf(S,3)+65)%91+32); //   Parses each 3-characters token into a number, using base 3,

                                           //  and make it a letter or a space

 return r;                                 //  return the result

}

Credits

• -14 bytes thanks to Kevin Cruijssen
  

R, 145 bytes

function(s,K=array(c(97:122,32),rep(3,3)))intToUtf8(K[matrix(arrayInd(match(c(utf8ToInt(s),32[!nchar(s)%%3]),K),dim(K))[,3:1],,3,byrow=T)[,3:1]])

Try it online!

I/O as strings; adds one space. The strange repetition of [,3:1] is because R's natural array indexing is somewhat different.

JavaScript (Node.js),  146 141 139  136 bytes

I/O is in lowercase.

s=>'931'.replace(/./g,d=>Buffer(s.length%3?s:s+0).map(c=>(o=(c>48?c-16:26)/d%3+o*3%27|0,++i)%3?0:(o+97)%123||32),i=o=0).split``.join``

Try it online!

Commented

s =>                       // s = input string

  '931'.replace(/./g, d => // for each digit d = 9, 3 and 1:

    Buffer(                //   create a buffer from:

      s.length % 3 ?       //     if the length of s is coprime with 3:

        s                  //       the original input string

      :                    //     else:

        s + 0              //       the input string + an extra '0'

    )                      //

    .map(c =>              //   for each ASCII code c from this Buffer:

      ( o =                //     update o:

        ( c > 48 ?         //       if c is neither a space nor the extra '0':

            c - 16         //         yield c - 16 (which gives 81 .. 106)

          :                //       else:

            26             //         this is the 26th character (space)

        ) / d % 3 +        //       divide by d and apply modulo 3

        o * 3 % 27 | 0,    //       add o * 3, apply modulo 27, coerce to integer

        ++i                //       increment i

      ) % 3 ?              //     if i mod 3 is not equal to 0:

        0                  //       yield 0 (NUL character)

      :                    //     else:

        (o + 97) % 123     //       convert o to the ASCII code of the output letter

        || 32              //       or force 32 (space) for the 26th character

    ),                     //   end of map()

    i = o = 0              //   start with i = o = 0

  ).split``.join``       // end of replace(); remove the NUL characters

SAS, 305 bytes

A hearty 'oof' for this SAS monstrosity. There's a lot of random string formatting that I thought I could avoid going into this; I'm sure there are better ways of doing some of this.

data;input n:&$99.;n=tranwrd(trim(n)," ","{");if mod(length(n),3)=0then n=cats(n,'{');f=n;l=length(n);array a(999);do i=1to l;v=rank(substr(n,i,1))-97;a{i}=int(v/9);a{i+l}=mod(int(v/3),3);a{i+l*2}=mod(v,3);end;f='';do i=1to l*3by 3;f=cats(f,byte(a{i}*9+a{i+1}*3+a{i+2}+97));end;f=tranwrd(f,"{"," ");cards;

Input is entered on newlines after the cards statement, like so:

data;input n:&$99.;n=tranwrd(trim(n)," ","{");if mod(length(n),3)=0then n=cats(n,'{');f=n;l=length(n);array a(999);do i=1to l;v=rank(substr(n,i,1))-97;a{i}=int(v/9);a{i+l}=mod(int(v/3),3);a{i+l*2}=mod(v,3);end;f='';do i=1to l*3by 3;f=cats(f,byte(a{i}*9+a{i+1}*3+a{i+2}+97));end;f=tranwrd(f,"{"," ");cards;

this is a trifid cipher

test

output

trifidcipher

Outputs a dataset containing the output in the variable f, along with a bunch of helper variables/array values.

Ungolfed/explanation:

data;

input n : & $99.; /* Read a line of input, maximum 99 characters */



n=tranwrd(trim(n)," ","{"); /* Replace spaces with '{' (this is the ASCII character following 'z', so it makes it easy to do byte conversions, and lets us not have to deal with spaces, which SAS does not like) */

if mod(length(n),3)=0then n=cats(n,'{'); /* If length of n is not coprime with 3, add an extra "space" to the end */



f=n; /* Set output = input, so that the string will have the same length */

l=length(n);    /* Get the length of the input */

array a(999);   /* Array of values to store intermediate results */



do i = 1 to l; /* For each character in the input... */

    v = rank(substr(n,i,1))-97; /* Get the value of the current character, from 0-26 */



    a{i}=int(v/9);          /* Get the table of the current character and store at appropriate index, from 0-2  */

    a{i+l}=mod(int(v/3),3); /* Get the row of the current character, from 0-2 */

    a{i+l*2}=mod(v,3);      /* Get the column of the current character, from 0-2  */

end;



f='';



do i = 1 to l*3 by 3; /* For each character in the output... */

    f=cats(f,byte(a{i}*9+a{i+1}*3+a{i+2}+97)); /* Convert values back from base 3 to base 10, and convert back into ASCII value */

end;



f = tranwrd(f,"{"," "); /* Replaces our "spaces" with actual spaces for final output */



/* Test cases */

cards;

this is a trifid cipher

test

output

trifidcipher