Krdytkyu

A famous definition which led to a completely new point of view is the definition of Schwartz distribution. It changed the understanding of what a "function" is, even among engineers.

Actually, the definition of function by Dirichlet in 19th century also clarified many things.

In set theory, definitely the notion of a Woodin cardinal.

First, it is not an entirely straightforward notion to guess. Significant large cardinals were up to that point defined as critical points of certain elementary embeddings. This is not the case here: Woodin cardinals need not be measurable. If $kappa$ is Woodin, then $V_kappa$ is a model of set theory where there is a proper class of strong cardinals. Woodinness requires more, namely, that these strong embeddings move some predicates correctly.

Second, the definition turned out to identify a pivotal point in inner model theory: for notions weaker than Woodinness, the corresponding canonical inner models carry $mathbfDelta^1_3$ well-orderings of the reals. This is no longer true once we have Woodin cardinals. This is closely tied up to the complexity of the comparison process: given two set models that look like initial segments of canonical inner models, how hard is to compare them to tell which one carries more information? For notions weaker than Woodinness, this process is carried out in an essentially linear fashion: disagreements between the models are witnessed by some measures, and repeatedly using these least measures to form ultrapowers eventually lines the models up: One of the iterates ends up as an initial segment of the other, and whichever one is longer comes from the model which originally has more information.With Woodin cardinals and beyond this process is no longer enough. Instead, comparisons sometimes need to retrace steps, and rather than a linear iteration, at the end we have tree-like structures. Identifying these crucial differences allowed us to develop inner model theory beyond this point. This in turn has led to many results and to the identification of deep connections between large cardinals and descriptive set theory. Literally, thanks to the presence of Woodin cardinals, the set-theoretic landscape grew and transformed significantly.

Third, Woodinness also turns out to be the notion needed to carry out certain forcing constructions. Some consistency results that were not expected now could be established, thanks to the identification of genuinely new forcing notions that use the Woodin cardinals in an essential way. Other constructions that were known from significant large cardinals were improved to their optimal form.

The story of how the notion of Woodinness was identified is actually quite nice. Kunen had used huge cardinals to prove the consistency of the existence of saturated ideals on $omega_1$. The embeddings witnessing hugeness end up lifting to the generic embeddings witnessing saturation in the appropriate forcing extension. In 1984, Foreman, Magidor and Shelah identified an entirely new way of finding models with saturated ideals. Their construction improved the large cardinal notion needed from hugeness to supercompactness. What is significant is that the generic embedding is no longer a lifting of an old genuine embedding. Indeed, their construction preserves $omega_1$. As a consequence of their results, in May of the same year, Woodin showed that supercompactness implies that all projective sets of reals are Lebesgue measurable (and more). Conversations between Shelah and Woodin quickly led to the realization that much weaker cardinals than supercompact sufficed for this result. Through a series of refinements, the notions of what we now call Shelah and Woodin cardinals were identified, with the latter being of precisely the right strength: all this happened while developments in determinacy and inner model theory showed the deep connections between these fields, and the pivotal role that Woodin cardinals played in this connection. This all happened quite quickly: by the time the Shelah-Woodin paper appeared in print, the importance of Woodin cardinals was already recognized.

MR1074499 (92m:03087). Shelah, Saharon; Woodin, Hugh. Large cardinals imply that every reasonably definable set of reals is Lebesgue measurable. Israel J. Math. 70 (1990), no. 3, 381–394.

A good example comes from the definition of manifolds, although it's less of a single definition and more of an evolving notion.

The background to modern differential geometry lies in trying understand cases where the parallel postulate fails. However, trying to rigorously pin down what sort of spaces to study is pretty tricky. It's only tangentially related, but there's a famous paper by Lakatos called "Proofs and Refutations" about the Euler characteristic on surfaces which gives insight as to why defining spaces precisely is a non-trivial problem.

Riemann had an intuitive notion of manifolds, and it's possible to see how his ideas evolved into the modern notion. It seems that Riemann's "Mannigfaltigkeit" had an element of "I know it when I see it." Similarly, Poincare didn't have a modern definition for manifold in Analysis Situs. The modern definition of a smooth manifold as a locally-Euclidean space wasn't introduced until 1912 by Hermann Weyl.
Even then, it wasn't clear until the Whitney proved the embedding theorem that the intrinsic definition was equivalent to submanifolds of Euclidean space.

That's still not the end of the story. If one tries to really pin down the definition, one realizes that there are at least three distinct categories of manifolds; smooth, piece-wise linear, and topological manifolds. The fact that these different types are not equivalent gives rise to a lot of research that is still continuing today.

In the comments I offered the "classical" example of Grothendieck's definition of a scheme. But for a more modern example: Fomin and Zelevinsky's definition of a cluster algebra is at first sight a rather ungainly thing, but turned out to be exactly what was needed to unify certain patterns that appeared in various hitherto unrelated areas of mathematics like discrete integrable systems, the representation theory of associative algebras, Poisson/symplectic geometry, etc.

My nominee is the definition of a topology. I don't know the history of the subject but my impression is that this was the result of a collective effort. The definition with collections of subsets closed under finite intersections and arbitrary unions is not the kind of thing one would get up one morning and decide to consider. It seems to me this was an example of category where the definition of morphisms was clear (continuous maps) but where finding the right notion of object was not obvious.

Perhaps a sign of a good definition is that it leads to other good definitions. Those could arise in order to remedy some flaws of the original one, like Grothendieck's topos. They could also arise as particular cases of the original definition like Schwartz's definition of the topology on $mathcal{D}$ underlying the notion of distribution mentioned in Alexandre's answer.

I would nominate the definition of probability theory in terms of measure theory, where a probability space is an abstract measure space, an event is a measurable subset, a random variable is a measurable function, and expectation is the Lebesgue integral. This is usually credited to a 1933 paper by Kolmogorov, though it seems that many of the ideas had previously been around. Here is a very interesting survey by Shafer and Vovk, entitled "The origins and legacy of Kolmogorov's Grundbegriffe."

People had been thinking about probability for a long time before that, but these definitions made it possible to place everything on a rigorous footing and exploit many key results from measure theory. Shafer and Vovk say that it took probability from a mathematical "pastime" to become a respected branch of pure mathematics.

(I am no expert historian so please feel free to edit with corrections, more background, further discussion, etc.)

I am surprised no-one has mentioned Weierstrass's $epsilon,delta$ definition of limits. This enabled mathematicians to rigorously reason about convergence and eliminate numerous apparent inconsistencies.

My interest in this is mainly psychological, namely, how the action of naming something somehow brings it into existence and organizes the world around it.

The power of naming...

J.-M. Souriau wrote a book meant to take anyone from the crib to relativistic cosmology and quantum statistics — that he unfortunately (predictably?) never got to really finish, but the draft is online. He starts by saying that all babies are (of course) natural born mathematicians, and everyone’s “first and sensual” abstraction is when they name that recurring warm blob “mommy”.

Surely that qualifies as “crucial to further understanding”; he argues that half math is more of the same, and tells stories illustrating the power of naming numbers, points, the unknown $(x)$, maps, variables, determinants, groups, morphisms, cohomologies... all would qualify as answers. It’s not just about coming up with epoch-making definitions, either: teaching students to define and name variables in their problems is also what we do.

We’ve all seen quotes to the effect that it’s all in the definitions. “Success stories” in that are largely the history of math, or at least one way to look at it. Specific recent ones that likely inspired the book (and made it to the MSC) would be symplectic manifold (1950), symplectic geometry (1953; earlier it was this), and moment(um) map (1967).

(I agree with others that this is a near-duplicate, and will transfer my answer there if needed.)

Well, this is basically the same answer as the one by Alexandre Eremenko, but here goes: The particular form of the definition of derivative is crucial for partial differential equations. Using weaker notions than classical derivatives leads to a whole new theory of solutions. While this is (part of) what the answer about distributions is about, the particular case of weak derivatives due to Sobolev predates distributions and is of prime importance to the field of PDEs. Also, weak derivatives allow for a rich theory of function spaces (Banach and Hilbert ones).

Euclid did not have real numbers. He could say that the length of one line segment goes into the length of another a certain number of times, or that the ratio of lengths of two segments was the same as that of two specified numbers ("numbers" were $2,3,4,5,ldots,$ and in particular $1$ was not a "number") or that the ratio of the length of the diagonal of a square to the side was not the ratio of any two numbers.

There is a much celebrated essay by James Wilkinson titled The Perfidious Polynomial that I have not brought myself to read beyond the first couple of pages because of the profound stupidity of what Wilkinson claims is the history of the development of number systems. He states with a straight face that negative numbers were introduced for the purpose of solving polynomial equations that had no positive solutions, that rational numbers were introduced after that for the purpose of solving yet more equations, that irrational numbers were then introduced for solving yet more, and finally complex numbers for the purpose of solving quadratic equations that lacked real solutions. Did Wilkinson never hear that Euclid proved the statement above about the ratio of diagonal to side of a square, but Euclid had never heard of negative numbers, and did not even have anything that we would consider a system of numbers that includes positive rational and irrational numbers? That imaginary numbers were used to find real solutions of third-degree polynomials? That negative numbers were introduced by accounts for their purposes? And that all of that obviously makes more sense and is more elegant and intellectually satisfying than the childish story he tells? In order to do what Wilkinson says was done, one would need certain definitions that were introduced only much later.