Krdytkyu

Does there exist a finite group $G$ and a normal subgroup $H$ of $G$ such that $lvertoperatorname{Aut}{(H)}rvert>lvertoperatorname{Aut}{(G)}rvert$?

A version of Robin Chapman's answer that might be easier to verify:

The additive group $H$ of a field of order $2^{n}$ has two nice types of automorphism: multiplication by a scalar, and the Frobenius automorphism. The semi-direct product $G$ is called $AΓL(1,2^{n})$. It has order $(2^{n})⋅(2^{n}−1)⋅(n) ≤ 2^{3n}$, and for $n ≥ 2$ it is its own automorphism group.

It has the additive group $H$ of the field as a normal subgroup. However, as just an additive group, $H$ has automorphism group all n×n matrices over the field with 2 elements, which has size $(2^{n}−1)⋅(2^{n}−2)⋅⋅⋅(2^{n}−2^{n−1}) ≥ 2^{(n−1)^{2}}$.

For large $n$, $|Aut(G)| = (2^{n})⋅(2^{n}−1)⋅(n) ≤ 2^{3n} ≤ 2^{(n−1)^{2}} ≤ (2^{n}−1)⋅(2^{n}−2)⋅⋅⋅(2^{n}−2^{n−1}) = |Aut(H)|$

In particular:

For $n = 4$, $G = AΓL(1,16)$, $Aut(G) = G$ has order $960$, and $Aut(H) = GL(4,2)$ has order $20160$.

For $n = 10$, $G = AΓL(1,1024)$, $Aut(G) = G$ has order $10475520$, and $Aut(H) = GL(10,2)$ has order $366440137299948128422802227200$.

In other words, $Aut(H)$ can be enormously bigger than $Aut(G)$.

This is reasonably important in finite group theory:

$G$ is a very rigid group with lots of structure. Because it contains all the "automorphisms" of the field, the group itself determines the field. An automorphism of the group will have to be an automorphism of the field, and we've already listed them all. Sometimes this expressed by saying the group G determines the geometry of the affine line on which it acts.

$H$ is a very floppy group to which you can do nearly anything. Without the maps encoding scalar multiplication, $H$ no longer remembers the field that defined it. It is just a vector space, and so instead of the (very few) field automorphisms, you are now allowed to use any vector space automorphism. $H$ has lost its structure.

I think $H$ is the canonical example of a horribly structureless group. Groups like $GL$ and $AΓL$ are pretty standard examples of groups with very clear structure. The symmetric group is very similar. Except for a few early cases, a symmetric group is its own automorphism group because it already contains within itself the set of points on which it is acting.

The quasidihedral group of order 16, QD16, has automorphism group of order 16, but it has Q8 (the quaternion group) as a normal subgroup, and | Aut(Q8) | = 24.

Steve

An elementary Abelian group can have a large automorphism group.
Let $H$ be a largish elementary Abelian group, and $G$
be an extension of $H$, say be a semidirect product with
a cyclic group acting faithfully on $H$. Then it is possible
for Aut$(G)$ to be smaller than Aut$(H)$.

Let $F_4 = Z/2[w]/w^2+w+1$ be the field of order 4.

Let $H = (F_4^+)^n = (Z/2)^{2n}$ for $n geq 2$.

Then $Aut(H) = GL(2n,F_2)$ which has order
$(4^n-1)(4^n-2)(4^n-4)...(4^n-2*4^(n-1))$

Let $G$ be $langle H,a : a^3 = 1, axa^{-1} = wx text{for} x in H rangle$. So $G$ is the semi-direct product of $H$ with $Z/3$, where the action is given by multiplication by $H$. An automorphism of $G$ must fix $H$ ($H$ consists of the set of elements of order 2). It must also send a to one of the 2*4^n elements of order 3. Let us consider only the automorphisms of $G$ which fix a (which is a subgroup of $Aut(G)$ of index $2*4^n$). These are automorphisms of H which commute with multiplication by w. Hence these are automorphisms of $F_4^n$ as an $F_4$ vector space, or elements of $GL(n,F_4)$. Hence
$|Aut(G)| = 2*4^n* (4^n-1)(4^n-4)...(4^n-4^(n-1))$.

For $n geq 2$, it is clear that $|Aut(H)| > |Aut(G)|$.

Problem 5

Also, you my find the answer at IMC 2008, Problem 5, day 1 (Already given)

EDITED: thanks to JackSchmidt