Under 1 year old:
Give them a box. They're free at the supermarket. Wrap it if you like, but it's not really necessary. We've given Theo a box every Christmas he's had so far. He always loves it.
This one will need to be used under adult supervision, but I reckon a pair of reading glasses from the $2 shop that your babe is allowed to pull off your face and play with would go down a treat. (I'll be suggesting many a thing from The $2 Shop throughout the series)
1-3 year old:
At The Warehouse (another cheap go-to for me) they have a set of the whole alphabet and numbers 0-9, set in squares roughly 20cm square, made of foam, with interlocking sides, selling for a whopping $6. We got Theo this for his second Christmas. He opened the packet on the spot, played with it all day, and still plays with it everyday a year later. A definite winner.
3-5 year old:
When my sister-in-law had her second baby, her first was about 3, and I sent over some learn to write books from the good old $2 Shop. You know, the type that you trace the letters, follow the lines with your pencil, all that type of thing. $2 per book, hours of activity (if they use pencil and you have an eraser). She asked for more! Kids this age just seem to love to write.
Hope that's helpful! Christmas on the Cheap will last a whole week and finish in time for you to have all your presents bought, wrapped and delivered in time (hopefully).