The game finished 0-0 after 90 minutes and went straight to spot-kicks.
Incredibly, Christian Eriksen missed the first penalty, before Luke Norris gave the hosts the advantage.
Dele Alli then scored his to pull things back, before Colchester went 2-1 up. Erik Lamela netted before Jevani Brown tried and failed to score a Panenka.
Son Heung-min scored before Lucas Moura missed for the Premier League side, as Tom Lapslie scored the decider, finishing the shootout 4-3.
It’s the second defeat in four days for Spurs, but this one is far more embarrassing than Saturday’s result at Leicester.
Mauricio Pochettino did shuffle his pack, but the team still had a good balance of youth and experience.
The likes of Alli, Eric Dier and Victor Wanyama all started, while youngsters Troy Parrott, Oliver Skipp and Japhet Tanganga also featured from kick-off.
