Heat oven to 180C/160C fan/gas 4. Grease and line a 20 x 30cm baking tray with baking parchment. Put the dates and milk in a small pan and bring to a simmer. Remove from the heat and set aside for 15 mins to cool.
Whizz the date mixture to a smooth purée in a food processor or blender, then scrape into a large mixing bowl. Tip in the butter, flour, brown sugar, baking powder, eggs, mixed spice and vanilla, and set aside while you prepare the apples.
Quarter and core the apples, then slice quite thinly, tossing in a little lemon juice as you go. Quickly beat together the cake ingredients with an electric whisk until smooth, then scrape into the baking tray. Arrange the apple slices, overlapping in rows, on top of the cake – you should be able to get 3 rows lengthways down the tray. Bake for 45-50 mins until a skewer poked into the centre of the cake comes out clean. Cool in the tray.
Put the toffees in a small pan with a good splash of milk and gently melt, stirring, until runny. To finish, dust with a little icing sugar, then drizzle sauce all over the cake. Cut into squares or slices to serve.