We study the assignment of flexible servers to stations in tandem lines with finite intermediate buffers and service times that are not necessarily exponentially distributed. More specifically, for systems with two stations and two servers, we show that the optimal server assignment policy is the same as the optimal policy for the corresponding Markovian system if the service times are deterministic and the buffer size is arbitrary or if the service times have an arbitrary distribution and the buffer size is zero. We also provide numerical results supporting the conjecture that the optimal server assignment policy of Markovian systems is also optimal for non-Markovian systems. For systems with two stations, three servers and buffer of size zero between the two stations, we specify the optimal policy when the service times are deterministic and we show that the structure of the optimal policy is of threshold type (as the one for the corresponding Markovian system) when the service times have an arbitrary distribution. Finally, we propose several heuristic server assignment policies and show that they yield near-optimal performance for different service time distributions.