Despite both being made of exclusively carbon atoms, brittle graphite, and sturdy diamonds have very different strengths. This physical property is all caused by the structure.
Diamonds have a giant covalent lattice structure, a regular pattern of strong covalent bonds in a tetrahedral shape. These strong bonds and strong shapes make Diamond a very sturdy structure. Its resistance to compression makes it useful in mining equipment.
In Graphite, carbon forms 2D layers in a tessellating hexagonal shape connected by covalent bonds. These layers are weakly bonded together by intermolecular bonds, which are more easily broken, making graphite very brittle. This makes graphite useful in pencils, the graphite breaking apart as you write, leaving your markings behind on the page.
Figure 1 - Shows how the rigid tetrahedral structure of a diamond is what makes it strong. And how the 2D hexagonal structure of graphite is what makes it weak.