They were developed to calculate ground and excited states of higher-dimensional systems originating in the one-dimensional matrix product states (equivalent to the density matrix renormalization group). We aim to explain their mathematical background rooted in maximizing quantum entanglement. We show the main principles in the simplest examples. Due to a so-called quantum-classical correspondence (imaginary time evolution), we analyze two identical representations of the Tensor Networks: 1) the vertex representation (commonly used in the quantum states) and 2) the weight representation (well-known from classical statistical mechanics). We explain two-dimensional classical spin models exhibiting equivalent partition functions. Although the equivalency of all observables and correlations for both representations, entanglement entropy is always lower for the vertex representation of the Tensor Networks for finite and infinite systems, irrespective of the spin-lattice model, the number of spin degrees of freedom, and the underlying lattice geometry. We consider various types of tensor networks to investigate why the discrepancy persists within all phase-transition types generated in a wide range of spin Hamiltonians."