When populations colonize new areas, both strong selection and strong drift can be experienced due to novel environments and small founding populations, respectively. Empirical studies have predominantly focused on the phenotype when assessing the role of selection, and limited neutral-loci when assessing founder-induced loss of diversity. Consequently, the extent to which processes interact to influence evolutionary trajectories is difficult to assess. Genomic-level approaches provide the opportunity to simultaneously consider these processes. Here, we examine the roles of selection and drift in shaping genomic diversity and divergence in historically documented sequential island colonizations by the silvereye (Zosterops lateralis). We provide the first empirical demonstration of the rapid appearance of highly diverged genomic regions following population founding, the position of which are highly idiosyncratic. As these regions rarely contained loci putatively under selection, it is most likely that these differences arise via the stochastic nature of the founding process. However, selection is required to explain rapid evolution of larger body size in insular silvereyes. Reconciling our genomic data with these phenotypic patterns suggests there may be many genomic routes to the island phenotype, which vary across populations. Finally, we show that accelerated divergence associated with multiple founding steps is the product of genome-wide rather than localized differences, and that diversity erodes due to loss of rare alleles. However, even multiple founder events do not result in divergence and diversity levels seen in evolutionary older subspecies, and therefore do not provide a shortcut to speciation as proposed by founder-effect speciation models.