Research on Routing Optimization in Satellite Internet Based on Deep Reinforcement Learning

Linhui WEI; Guowen LIU; Yu LIU; Yumei WANG

doi:10.11959/j.issn.2096-8930.2022033

您当前的位置：

首页 >

文章列表页 >

Research on Routing Optimization in Satellite Internet Based on Deep Reinforcement Learning

Special Issue: World-Earth Integration Software-Defined Network | 更新时间：2024-06-03

- Research on Routing Optimization in Satellite Internet Based on Deep Reinforcement Learning
- Space-Integrated-Ground Information Networks Vol. 3, Issue 3, Pages: 65-71(2022)
- 作者机构：
  
  1. 北京邮电大学人工智能学院，北京 100876
  2. 鹏城实验室，广东深圳 518000
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-8930.2022033
  CLC： TN927.2
- Published Online：2022-09，
  
  Published：20 September 2022
- 稿件说明：
移动端阅览
Linhui WEI, Guowen LIU, Yu LIU, et al. Research on Routing Optimization in Satellite Internet Based on Deep Reinforcement Learning[J]. Space-integrated-ground information networks, 2022, 3(3): 65-71.
DOI：

Linhui WEI, Guowen LIU, Yu LIU, et al. Research on Routing Optimization in Satellite Internet Based on Deep Reinforcement Learning[J]. Space-integrated-ground information networks, 2022, 3(3): 65-71. DOI： 10.11959/j.issn.2096-8930.2022033.

摘要

随着卫星通信技术的飞速发展，卫星互联网成为6G网络实现全球覆盖、全时接入、全场景服务的核心关键技术。卫星网络的高动态性及有限的卫星容量，导致面临以异构网络管理、动态资源分配为代表的一系列管控挑战。由于机器学习技术在网络设计等方面具有显著优势，因此提出软件定义的卫星互联网智能化架构。针对卫星互联网的智能路由问题，利用基于双延迟深度确定性策略梯度的深度强化学习算法，解决网络的实时路由优化问题。实验结果表明，TD3算法相较于DDPG算法，平均网络时延降低了19.19%。

Abstract

With the rapid development of satellite communication

the satellite internet is one of the core technologies of 6G network to realize global coverage

full-time access and full scene service.The high dynamics and limited capacity of satellite network lead to a series of management and control challenges such as heterogeneous network management

dynamic resource allocation and so on.Since the machine learning-based technologies have strength in network design

the intelligent architecture of software-defi ned satellite internet was put forward.In view of the intelligent routing in satellite internet

and leverages the deep reinforcement algorithm based on double delayed deep deterministic policy gradient (TD3) to solve the network routing optimization problem.The experimental results showed that compared with DDPG algorithm

the TD3 algorithm reduced the delay by 19.19%.

关键词

Keywords

references

吴巍 . 卫星互联网发展综述 [J ] . 天地一体化信息网络 , 2020 , 1 ( 1 ): 1 - 16 .

WU W . Survey on the development of space-integrated-ground information network [J ] . Space-Integrated-Ground Information Networks , 2020 , 1 ( 1 ): 1 - 16 .

KREUTZ D , RAMOS F M V , VERÍSSIMO P E , , et al . Softwaredefined networking,a comprehensive survey [J ] . Proceedings of the IEEE , 2015 , 103 ( 1 ): 14 - 76 .

TANG Z , ZHAO B K , YU W R , et al . Software defined satellite networks,benefits and challenges [C ] // Proceedings of 2014 IEEE Computers,Communications and IT Applications Conference . Piscataway,IEEE Press , 2014 : 127 - 132 .

FORTZ B , THORUP M . Internet traffic engineering by optimizing OSPF weights [C ] // Proceedings of IEEE INFOCOM 2000.Conference on Computer Communications.Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat.No.00CH37064) . Piscataway,IEEE Press , 2000 : 519 - 528 .

XIE J F , YU F R , HUANG T , et al . A survey of machine learning techniques applied to software defined networking (SDN),research issues and challenges [J ] . IEEE Communications Surveys ＆ Tutorials , 2019 , 21 ( 1 ): 393 - 430 .

WANG M W , CUI Y , WANG X , et al . Machine learning for networking,workflow,advances and opportunities [J ] . IEEE Network , 2018 , 32 ( 2 ): 92 - 99 .

ARULKUMARAN K , DEISENROTH M P , BRUNDAGE M , et al . Deep reinforcement learning,a brief survey [J ] . IEEE Signal Processing Magazine , 2017 , 34 ( 6 ): 26 - 38 .

YAO H P , WANG L Y , WANG X D , et al . The space-terrestrial integrated network,an overview [J ] . IEEE Communications Magazine , 2018 , 56 ( 9 ): 178 - 185 .

安建平 , 李建国 , 于季弘 , 等 . 空天通信网络关键技术综述 [J ] . 电子学报 , 2022 , 50 ( 2 ): 470 - 479 .

AN J P , LI J G , YU J H , et al . Key technologies of space-air-ground communication networks,a survey [J ] . Acta Electronica Sinica , 2022 , 50 ( 2 ): 470 - 479 .

SHI Y P , LIU J J , FADLULLAH Z M , et al . Cross-layer data delivery in satellite-aerial-terrestrial communication [J ] . IEEE Wireless Communications , 2018 , 25 ( 3 ): 138 - 143 .

YAO S , GUAN J F , YAN Z W , et al . SI-STIN,a smart identifier framework for space and terrestrial integrated network [J ] . IEEE Network , 2019 , 33 ( 1 ): 8 - 14 .

徐晖 , 孙韶辉 . 面向6G的天地一体化信息网络架构研究 [J ] . 天地一体化信息网络 , 2021 , 2 ( 4 ): 2 - 9 .

XU H , SUN S H . Research on network architecture for the spaceintegrated-ground information network in 6G [J ] . Space-IntegratedGround Information Networks , 2021 , 2 ( 4 ): 2 - 9 .

BI Y G , HAN G J , XU S , et al . Software defined space-terrestrial integrated networks,architecture,challenges,and solutions [J ] . IEEE Network , 2019 , 33 ( 1 ): 22 - 28 .

ZHANG N , ZHANG S , YANG P , et al . Software defined space-airground integrated vehicular networks,challenges and solutions [J ] . IEEE Communications Magazine , 2017 , 55 ( 7 ): 101 - 109 .

杨丹 , 刘江 , 张然 , 等 . 基于SDN的卫星通信网络,现状、机遇与挑战 [J ] . 天地一体化信息网络 , 2020 ( 2 ): 34 - 41 .

YANG D , LIU J , ZHANG R , et al . SDN-based satellite networks:progress,opportunities and challenges [J ] . Space-Integrated-Ground Information Networks , 2020 ( 2 ): 34 - 41 .

MESTRES A , RODRIGUEZ-NATAL A ,, CARNER J , et al . Knowledge-defined networking [J ] . ACM SIGCOMM Computer Communication Review , 2017 , 47 ( 3 ): 2 - 10 .

STAMPA G , ARIAS M , SANCHEZ-CHARLES D , et al . A deepreinforcement learning approach for software-defined networking routing optimization [EB ] . 2017 .

CHEN J , XIAO Z W , XING H L , et al . STDPG,a spatiotemporal deterministic policy gradient agent for dynamic routing in SDN [C ] // Proceedings of ICC 2020-2020 IEEE International Conference on Communications . Piscataway,IEEE Press , 2020 : 1 - 6 .

HUANG X H , YUAN T T , QIAO G H , et al . Deep reinforcement learning for multimedia traffic control in software defined networking [J ] . IEEE Network , 2018 , 32 ( 6 ): 35 - 41 .

TU Z , ZHOU H C , LI K , et al . A routing optimization method for software-defined SGIN based on deep reinforcement learning [C ] // Proceedings of 2019 IEEE Globecom Workshops . Piscataway,IEEE Press , 2019 : 1 - 6 .

SHI X J , REN P Y , DU Q H . Reinforcement learning routing in space-air-ground integrated networks [C ] // Proceedings of 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP) . Piscataway,IEEE Press , 2021 : 1 - 6 .

ZUO P L , WANG C , YAO Z , et al . An intelligent routing algorithm for LEO satellites based on deep reinforcement learning [C ] // Proceedings of 2021 IEEE 94th Vehicular Technology Conference . Piscataway:IEEE Press , 2021 : 1 - 5 .

李新桐 , 张亚生 . 一种适用于低轨卫星的SDN网络人工智能路由方法 [J ] . 电子测量技术 , 2020 , 43 ( 22 ): 109 - 114 .

LI X T , ZHANG Y S . Artificial intelligence routing method for SDN network suitable for LEO satellites [J ] . Electronic Measurement Technology , 2020 , 43 ( 22 ): 109 - 114 .

Views

1232

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Design of Satellite Internet Defense System Based on Mobile Target Defense

Unified Carrier Regulation and Management Technology for Multiple Baseband Systems in High Throughput Satellite Network

Satellite Internet：Application and Development

Dual-Mode Communication Terminal Technologies Based on LEO Satellite Internet

Analysis of Satellite Internet Security Risks and Countermeasures

Related Author

XU Ruirong

LI Yan

ZHAO Yiyun

WANG Yixuan

SHAN Chao

MA Shaoxuan

GU Xin

WANG Niwei

Related Institution

SINOPEC Shengli Oil-Field Electric Power Branch

China Satellite Communications Co., Ltd.

China Satellite Network Innovation Co., Ltd.

CETC Cyberspace Security Technology Co., Ltd.

Intelligent Network Research Institute Zhejiang Lab

⁰