Peel tamarillos, then slice or chop them. Simmer them in a syrup made from 1/2 cup sugar to 3/4 or 1 cup of water for 5 – 10 minutes until tender, then stand the pot in cold water to cool the fruit and syrup.
Meantime, cut unfilled sponge in cubes and arrange in individual dishes or one larger bowl. Sprinkle the sponge with sherry, then spoon some cooled, stewed tamarillos and syrup over the sponge, then cover with a layer of vanilla custard. Top with a little lightly whipped cream and toasted, slivered almonds before serving.
Variation: If you are going to eat the trifle within a couple of hours, you can replace the custard with instant (vanilla, lemon or butterscotch) pudding.