Federated Learning is an approach to distributed machine learning that enables collaborative model training on end devices. FL enhances privacy as devices only share local model parameters instead of raw data with a central server. However, the central server or eavesdroppers could extract sensitive information from these shared parameters. This issue is crucial in applications like speech emotion recognition (SER) that deal with personal voice data. To address this, we propose Optimized Paillier Homomorphic Encryption (OPHE) for SER applications in FL. Paillier homomorphic encryption enables computations on ciphertext, preserving privacy but with high computation and communication overhead. The proposed OPHE method can reduce this overhead by combing Paillier homomorphic encryption with pruning. So, we employ OPHE in one of the use cases of a large research project (DAIS) funded by the European Commission using a public SER dataset.