Indeed, in 10BaseT/100BaseT only 2 pairs are used (the orange and green in positions 1, 2, 3, and 6 if you use the T568A standard). From 1000BaseT onwards, all four pairs are used.

The standard PoE only uses two pairs but it depends on the type. If it is Mode A, it also uses the same pairs 1-2, 3-6 as the 10/100 so you might only need the 2 pairs. If it is Mode B, it uses the alternating pairs for power and you would need all eight. If it is 4PPoE mode, you need all 4 pairs for power (usually for cameras with high consumption).

For 1Gb and above, you will always need all 8 in any case.