The Information Bottleneck (IB) framework provides a principled and broadly applicable approach for studying efficient compressed representations in artificial and biological systems. However, a comprehensive mathematical understanding of the optimal IB representations and the structural phase transitions they undergo via deterministic annealing exists only in a few limited cases. Here, we address the case of symbolic, or discrete, representations, which is particularly relevant to the emergence of language and abstract representations more generally. We characterize the structural changes in the IB representations as they evolve via a deterministic annealing process; derive an algorithm for finding critical points; and explore numerically the types of bifurcations and related phenomena that occur in IB. This work extends the theoretical grounds for understanding optimal representations within the IB framework.