浏览全部资源
扫码关注微信
Published： 16 July 2024 ，
移动端阅览
黄文柯， 叶茫， 杜博. 2024. 自适应异构联邦学习. 中国图象图形学报， 29(07):1849-1860
Huang Wenke， Ye Mang， Du Bo. 2024. Adaptive heterogeneous federated learning. Journal of Image and Graphics， 29(07):1849-1860
目的
2
模型异构联邦学习由于允许参与者在不损害隐私的情况下独立设计其独特模型而受到越来越多的关注。现有的方法通常依赖于公共共享的相关数据或全局模型进行通信，极大地限制了适用性。且每个参与者的私有数据通常以不同的分布收集，导致数据异构问题。为了同时处理模型异构和数据异构，本文提出了一种新颖的自适应异构联邦学习方法。
方法
2
给定一个随机生成的输入信号（例如，随机噪声），自适应异构联邦学习直接通过对齐输出逻辑层分布来实现异构模型之间的通信，实现协作知识共享。主要优势是在不依赖额外相关数据收集或共享模型设计的情况下解决了模型异构问题。为了进一步解决数据异构问题，本文提出了在模型和样本层面上进行自适应权重更新。因此，自适应异构联邦学习（adaptive heteogeneous federated learning，AHF）允许参与者通过模型输出在无关数据上的差异和强调“有意义”的样本来学习丰富多样的知识。
结果
2
通过在不同的联邦学习任务上使用随机噪声输入进行通信，进行了广泛的实验，显示出比竞争方法更高的域内精确度和更好的跨域泛化性能。
结论
2
本文方法提供了一个简单而有效的基准，为异构联邦学习的未来发展奠定基础。
Objective
2
The current development of deep learning has caused significant changes in numerous research fields and has had profound impacts on every aspect of societal and industrial sectors， including computer vision， natural language processing， multi-modal learning， and medical analysis. The success of deep learning heavily relies on large-scale data. However， the public and scientific communities have become increasingly aware of the need for data privacy. In the real world， data are commonly distributed among different entities such as edge devices and companies. With the increasing emphasis on data sensitivity， strict legislation has been proposed to govern data collection and utilization. Thus， the traditional centralized training model， which requires data aggregation， is unusable in the practical setting. In response to such real-world challenges， federated learning （FL） has emerged as a popular research field because it can train a global model for different participants without centralizing data owned by the distributed parties. FL is a privacy-preserving multiparty collaboration model that adheres to privacy protocols without data leakage. Typically， FL requires clients to share a global model architecture for the central server to aggregate parameters from participants and then redistributes the global model （averaged parameters）. However， this prerequisite largely restricts the flexibility of the client model architecture. In recent years， the concept of objective model heterogeneous FL has garnered substantial attention because it allows participants to independently design unique models in FL without compromising privacy. Specifically， participants may need to design special model architecture to ease the communication burden or refuse to share the same architecture due to intellectual property concerns. However， existing methods often rely on publicly shared related data or a global model for communication， limiting their applicability. In addition， FL is proposed to handle privacy concerns in the distributed learning environment. A pioneering FL method trains a global model by aggregating local model parameters. However， its performance is impeded due to decentralized data， which results in non-i.i.d distribution （called data heterogeneity）. Each participant optimizes toward the local empirical risk minimum， which is inconsistent with the global direction. Therefore， the average global model has a slow convergence speed and achieves limited performance improvement.
Method
2
Model heterogeneity largely impedes the local model section flexibility， and data heterogeneity hinders federated performance. To address model and data heterogeneity， this paper introduces a groundbreaking approach called adaptive heterogeneous federated （AHF） learning， which employs a unique strategy by utilizing a randomly generated input signal， such as random noise and public unrelated samples， to facilitate direct communication among heterogeneous model architectures. This task is achieved by aligning the output logit distributions， fostering collaborative knowledge sharing among participants. The primary advantage of AHF is its ability to address model heterogeneity without depending on additional related data collection or shared model design. To further enhance AHF’s effectiveness in handling data heterogeneity， the paper proposes adaptive weight updating on both model and sample levels， which enables AHF participants to acquire rich and diverse knowledge by leveraging dissimilarities in model output on unrelated data while emphasizing the importance of meaningful samples.
Result
2
Empirical validation of the proposed AHF method is conducted through a meticulous series of extensive empirical experiments. Random noise inputs are employed in the context of two distinct federated learning tasks： Digits and Office-Caltech scenarios. Specifically， our solution presents the stable generalization performance on the more challenging scenario， Office-Caltech. Notably， when a larger domain gap exists among private data， AHF achieves higher overall generalization performance on these different unrelated data samples and obtains stable improvements on most unseen private data. By contrast， competing methods achieve limited generalization performance in the Office-Caltech scenario. The empirical findings validate our solution’s ability， showcasing a marked improvement in within-domain accuracy and demonstrating superior cross-domain generalization performance compared with existing methodologies.
Conclusion
2
In summary， the AHF learning method， as extensively examined in this thorough investigation， not only presents a straightforward yet remarkably efficient foundation for future progress in the domain of federated learning but also emerges as a transformative paradigm in comprehensively addressing model and data heterogeneity. AHF not only lays the groundwork for more resilient and adaptable FL models but also serves as a guide for the transformation of collaborative knowledge sharing in the upcoming era of machine learning. Studying AHF is more than an exploration of an innovative FL methodology； it provides numerous opportunities that arise given the complexities of model and data heterogeneity in the development of machine learning models.
联邦学习（FL）模型异构数据异构随机噪声异构联邦学习
federated learning （FL）model heterogeneitydata heterogeneityrandom noiseheterogeneous federal learning
Arivazhagan M G， Aggarwal V， Singh A K and Choudhary S. 2019. Federated learning with personalization layers ［EB/OL］. ［2022-05-21］. https://arxiv.org/pdf/1912.00818.pdfhttps://arxiv.org/pdf/1912.00818.pdf
Chang H Y， Shejwalkar V， Shokri R and Houmansadr A. 2019. Cronus： robust and heterogeneous collaborative learning with black-box knowledge transfer ［EB/OL］. ［2022-05-21］. https://arxiv.org/pdf/1912.11279.pdfhttps://arxiv.org/pdf/1912.11279.pdf
Dinh C T， Tran N H and Nguyen T D. 2020. Personalized federated learning with Moreau envelopes//Proceedings of the 34th Conference on Neural Information Processing Systems. Vancouver， Canada： Curran Associates Inc.： 21394-21405
Fallah A， Mokhtari A and Ozdaglar A. 2020. Personalized federated learning with theoretical guarantees： a model-agnostic meta-learning approach//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver， Canada： Curran Associates Inc.： 3557-3568 ［DOI： 10.5555/3495724.3496024http://dx.doi.org/10.5555/3495724.3496024］
Fang X W and Ye M. 2022. Robust federated learning with noisy and heterogeneous clients//Proceedings of 2022 IEEE Conference on Computer Vision and Pattern Recognition. New Orleans， USA： IEEE.
Guo X， Xing P W， Feng S W， Li B A and Miao C Y. 2020. Federated learning with diversified preference for humor recognition//International Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with IJCAI. Yokohama， Japan
Hard A， Rao K， Mathews R， Beaufays F， Augenstein S， Eichner H， Kiddon C and Ramage D. 2018. Federated learning for mobile keyboard prediction ［EB/OL］. ［2023-05-08］. https://arxiv.org/pdf/1811.03604.pdfhttps://arxiv.org/pdf/1811.03604.pdf
He K M， Zhang X Y， Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas， USA： IEEE： 770-778 ［DOI： 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90］
Howard A G， Zhu M L， Chen B， Kalenichenko D， Wang W J， Weyand T， Andreetto M and Adam H. 2018. MobileNets： efficient convolutional neural networks for mobile vision applications ［EB/OL］. ［2023-05-08］. https://arxiv.org.pdf.1704.04861.pdfhttps://arxiv.org.pdf.1704.04861.pdf
Huang W K， Ye M and Du B. 2022. Learn from others and be yourself in heterogeneous federated learning//Proceedings of 2022 IEEE Conference on Computer Vision and Pattern Recognition. New Orleans， USA： IEEE.
Hull J J. 1994. A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence： 550-554.
Jiang Y， Konečný J， Rush K and Kannan S. 2019. Improving federated learning personalization via model agnostic meta learning ［EB/OL］. ［2023-05-08］. https://arxiv.org/pdf/1909.12488pdfhttps://arxiv.org/pdf/1909.12488pdf
Ju C， Zhao R H， Sun J C， Wei X G， Zhao B， Liu Y， Li H S， Chen T J， Zhang X W and Gao D S. 2020. Privacy-preserving technology to help millions of people： federated prediction model for stroke prevention. ［EB/OL］. ［2023-05-08］. https://arxiv.org.pdf.2006.10517.pdfhttps://arxiv.org.pdf.2006.10517.pdf
Kairouz P， Brendan McMahan H， Avent B， Bellet A， Bennis M， Bhagoji A N， Bonawit K， Charles Z， Cormode G， Cummings R， D’Oliveira R G L， Eichner H， El Rouayheb S， Evans D， Gardner J， Garrett Z， Gascón A， Ghazi B， Gibbons P B， Gruteser M， Harchaoui Z， He C Y， He L， Huo Z Y， Hutchinson B， Hsu J， Jaggi M， Javidi T， Joshi G， Khodak M， Konecný J， Korolova A， Koushanfar F， Koyejo S， Lepoint T， Liu Y， Mittal P， Mohri M， Nock R， Özgür A， Pagh R， Qi H， Ramage D， Raskar R， Raykova M， Song D， Song W K， Stich S U， Sun Z T， Suresh A T， Tramèr F， Vepakomma P， Wang J Y， Xiong L， Xu Z， Yang Q， Yu F X， Yu H and Zhao S. 2019. Advances and open problems in federated learning ［EB/OL］. ［2023-05-08］. https://arxiv.org/pdf/1912.04977.pdfhttps://arxiv.org/pdf/1912.04977.pdf
Kairouz P， McMahan H B， Avent B， Bellet A， Bennis M， Bhagoji A N， Bonawitz K， Charles Z， Cormode G and Cummings R. 2019. Advances and open problems in federated learning［EB/OL］. ［2023-05-08］. https://arxiv.org.pdf.1912.04977.pdfhttps://arxiv.org.pdf.1912.04977.pdf
Kingma D P and Ba J. 2014. Adam： a method for stochastic optimization ［EB/OL］. ［2023-05-08］. https://arxiv.org/pdf/1412.6980.pdfhttps://arxiv.org/pdf/1412.6980.pdf
LeCun Y， Bottou L， Bengio Y and Haffner P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE， 86（11）： 2278-2324 ［DOI： 10.1109/5.726791http://dx.doi.org/10.1109/5.726791］
Li T， Sahu A K， Zaheer M， Sanjabi M， Talwalkar A and Smith V. 2018. Federated optimization in heterogeneous networks ［EB/OL］. ［2023-05-08］. https://arxiv.org/pdf/1812.06127.pdfhttps://arxiv.org/pdf/1812.06127.pdf
Li D L and Wang J P. 2019. FedMD： heterogenous federated learning via model distillation//Proceedings of International Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with NeurIPS. Canada： ［s.n.］：#03581
Liang P P， Liu T， Liu Z Y， Allen N B， Auerbach R P， Brent D， Salakhutdinov R and Morency Louis-P. 2019. Think locally， act globally： federated learning with local and global representations//Proceedings of International Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with NeurIPS. Canada： ［s.n.］
Lin T， Kong L J， Stich S U and Jaggi M. 2019. Ensemble distillation for robust model fusion in federated learning//Proceedings of Neural Information Processing Systems.Canada： ［s.n.］
Liu L M， Zong J X， Xiao Z J， Lan H and Qu H C. 2022. Cross-consistent semantic segmentation algorithm based on manifold regularization. Journal of Image and Graphics， 27（12）： 3542-3552
刘腊梅， 宗佳旭， 肖振久， 兰海， 曲海成. 2022. 流形正则化的交叉一致性语义分割算法. 中国图象图形学报， 27（12）： 3542-3552 ［DOI： 10.11834/jig.210571http://dx.doi.org/10.11834/jig.210571］
McMahan B， Moore E， Ramage D， Hampson S and Arcas B A Y. 2017. Communication-efficient learning of deep networks from decentralized data//Proceedings of the 20th International Conference on Artificial Intelligence and Statistic. ［s.l.］： PMLR： 1273-1282
Netzer Y， Wang T， Coates A， Bissacco A， Wu B and Ng A Y. 2011. Reading digits in natural images with unsupervised feature learning//Proceedings of Annual Conference on Neural Information Processing Systems. ［s.l.］： NeurIPSW
Peterson D， Kanani P and Marathe V J. 2019. Private federated learning with domain adaptation ［EB/OL］. ［2023-05-08］. https://arxiv.org/pdf/1912.06733.pdfhttps://arxiv.org/pdf/1912.06733.pdf
Roy P， Ghosh S， Bhattacharya S and Pal U. 2018. Effects of degradations on deep neural network architectures ［EB/OL］. ［2023-05-08］. https://arxiv.org/pdf/1807.10108.pdfhttps://arxiv.org/pdf/1807.10108.pdf
Russakovsky O， Deng J， Su H， Krause J， Satheesh S， Ma S， Huang Z H， Karpathy A， Khosla A， Bernstein M， Alexander C， Berg A C and Li F F. 2015. ImageNet large scale visual recognition challenge. International Journal of Computer Vision， 115（3）： 211-252 ［DOI： 10.1007/s11263-015-0816-yhttp://dx.doi.org/10.1007/s11263-015-0816-y］
Sattler F， Marban A， Rischke R and Samek W. 2020. Communication-efficient federated distillation ［EB/OL］. ［2023-05-08］. https://arxiv.org/pdf/2012.00632.pdfhttps://arxiv.org/pdf/2012.00632.pdf
Shen T， Zhang J， Jia X K， Zhang F D， Huang G， Zhou P， Wu F and Wu C. 2020. Federated mutual learning ［EB/OL］. ［2023-05-08］. https：//arxiv.org.pdf.2006.16765.pdfhttps://arxiv.org.pdf.2006.16765.pdf
Shoham N， Avidor T， Keren A， Israel N， Benditkis D， Mor-Yosef L and Zeitak I. 2019. Overcoming forgetting in federated learning on non-IID data//Proceedings of International Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with NeurIPS. Canada： ［s.n.］
Singhal K， Sidahmed H， Garrett Z， Wu S S， Rush K and Prakash S. 2021. Federated reconstruction： partially local federated learning//Proceedings of International Conference on Machine Learning.Virtual Online： ICML.
Sun L C and Lyu L J. 2020. Federated model distillation with noise-free differential privacy. ［EB/OL］. ［2023-05-08］. https://arxiv.org.pdf.2009.05537.pdfhttps://arxiv.org.pdf.2009.05537.pdf
Tan M X and Le Q. 2019. EfficientNet： rethinking model scaling for convolutional neural networks//Proceedings of International Conference on Machine Learning. Virtual Online： ICML：6105-6114
Virtual online： ICLR： 1-19
Xiao Z J， Zong J X， Lan H， Wei X and Tang X L. 2022. Image semantic segmentation based on manifold regularization constraint. Journal of Image and Graphics， 27（4）： 1204-1215
肖振久， 宗佳旭， 兰海， 魏宪， 唐晓亮. 2022. 流形正则化约束的图像语义分割. 中国图象图形学报， 27（4）： 1204-1215 ［DOI： 10.11834/jig.200527http://dx.doi.org/10.11834/jig.200527］
Yang Q， Liu Y， Chen T J and Tong Y X. 2019. Federated machine learning： concept and applications， ACM Transactions on Intelligent Systems and Technology， 2019： #3298981［DOI： 10.1145/3298981http://dx.doi.org/10.1145/3298981］
Yoon T， Shin S M， Hwang S J and Yang E H. 2021. FEDMIX： approximation of mixup under mean augmented federated learning//Proceedings of International Conference on Learning Representations.
Zhang M， Sapra K， Fidler S， Yeung S and Alvarez J M. 2021. Personalized federated learning with first order model optimization//Proceedings of International Conference on Learning Representations. Virtual Online： ICLR.
相关作者
相关机构
京公网安备11010802024621