Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning

Siyue CHENG; Haoran LI; Weigang BAI; Di ZHOU; Yan ZHU

doi:10.11959/j.issn.2096-8930.2023002

您当前的位置：

首页 >

文章列表页 >

Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning

Special Issue: Key Technologies for Networking and TT＆C in Large Scale Constellations | 更新时间：2024-06-03

- Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning
- Space-Integrated-Ground Information Networks Vol. 4, Issue 1, Pages: 12-22(2023)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-8930.2023002
  CLC： V19
- Published Online：2023-03，
  
  Published：20 March 2023
- 稿件说明：
移动端阅览
Siyue CHENG, Haoran LI, Weigang BAI, et al. Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning[J]. Space-integrated-ground information networks, 2023, 4(1): 12-22.
DOI：

Siyue CHENG, Haoran LI, Weigang BAI, et al. Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning[J]. Space-integrated-ground information networks, 2023, 4(1): 12-22. DOI： 10.11959/j.issn.2096-8930.2023002.

摘要

随着卫星通信技术的发展，星座规模的不断扩大，测运控一体化成为主流趋势。星座规模大、调度对象多、复杂操作联合控制给卫星网络测运控一体化资源调度带来巨大的挑战。受制于调度算法求解效率低、约束复杂等问题，传统的测运控资源调度技术采用提前上注测控指令，按照固定部署执行任务，难以满足突发事件与紧急任务的调度需求。因此，提出一种基于多智能体演员-评判家确定性策略梯度算法的测运控一体化资源调度方法，采用集中式训练和分布式执行的方法，建立测运控一体化任务的多智能体模型，通过分析邻居智能体局部信息计算调度策略，提高任务的响应速度。依据测运控一体化资源调度问题中的模型和约束，选择影响意义大、可解释的约束，建立多智能体资源调度强化学习模型，并进行仿真测试。测试结果显示，该方法的任务收益较传统方法提高22%。

Abstract

With the development of satellite communication technology and the continuous expansion of the constellation scale

the integration of TT＆amp;C and observation technology has become the mainstream trend.The large constellation scale

many scheduling objects and complex operation joint control bring great challenges to the integrated resource scheduling of satellite network TT＆amp;C and observation.Subject to the low solution effi ciency and complex constraints of scheduling algorithms

the traditional TT＆amp;C resource scheduling technology adopts the advance injection TT＆amp;C instructions to perform tasks according to the fi xed deployment

which is diffi cult to meet the scheduling needs of emergencies and emergency tasks.Therefore

a kind of resource scheduling method based on multi-agent actor-Agent Actor-Critic Deterministic Policy Gradient Algorithms (MADDPG) was presented.With centralized training and distributed execution

the multi-agent model of integrated task of TT＆amp;C and observation was established.By analyzed the scheduling strategy of neighbor agent

the response speed of local information was improved.According to the model and constraints in the integrated resource scheduling problem of TT＆amp;C and observation

selected signifi cant and interpretable constraints

then established the multi-agent resource scheduling reinforcement learning model

and carried on the simulation test.The simulation results showed that the task benefi t of this method was 22% higher than the traditional method.

关键词

Keywords

references

张威 , 吴涛 , 马宏 , 等 . 智能一体化航天测运控网络发展探析 [J ] . 天地一体化信息网络 , 2021 , 2 ( 2 ): 82 - 89 .

ZHANG W , WU T , MA H , et al . Discussion on the development of integrated and intelligent space TTC ＆ OC network [J ] . SpaceIntegrated-Ground Information Networks , 2021 , 2 ( 2 ): 82 - 89 .

GABREL V . Strengthened 0-1 linear formulation for the daily satellite mission planning [J ] . Journal of Combinatorial Optimization , 2006 , 11 ( 3 ): 341 - 346 .

王海波 , 徐敏强 , 王日新 , 等 . 对地观测小卫星星座长期任务规划求解技术 [J ] . 系统工程与电子技术 , 2011 , 33 ( 6 ): 1293 - 1298 .

WANG H B , XU M Q , WANG R X , et al . Long-term acquisition plan method for small satellites constellation [J ] . Systems Engineering and Electronics , 2011 , 33 ( 6 ): 1293 - 1298 .

王沛 . 基于分支定价的多星多站集成调度方法研究 [D ] . 长沙:国防科学技术大学 , 2011 .

WANG P . Research on branch-and-price based multi-satellite multi-station integrated scheduling method [D ] . Changsha:National University of Defense Technology , 2011 .

徐小辉 , 胡绍林 , 郭小红 , 等 . 航天器测控资源调度模型及算法 [J ] . 中国空间科学技术 , 2014 , 34 ( 3 ): 32 - 37 .

XU X H , HU S L , GUO X H , et al . Spacecraft TT ＆ C scheduling models and algorithms [J ] . Chinese Space Science and Technology , 2014 , 34 ( 3 ): 32 - 37 .

王远振 , 赵坚 , 聂成 . 多卫星—地面站系统的Petri网模型研究 [J ] . 空军工程大学学报(自然科学版) , 2003 , 4 ( 2 ): 7 - 11 .

WANG Y Z , ZHAO J , NIE C . Study on Petri net model for multisatellites-ground station system [J ] . Journal of Air Force Engineering University (Natural Science Edition) , 2003 , 4 ( 2 ): 7 - 11 .

ZHANG N , FENG Z R , FENG Y J , et al . An optimization model for multisatellite resources scheduling [C ] // Proceedings of 2006 6th World Congress on Intelligent Control and Automation . Piscataway:IEEE Press , 2006 : 7400 - 7404 .

ZHANG Z J , HU F N , ZHANG N . Ant colony algorithm for satellite control resource scheduling problem [J ] . Applied Intelligence , 2018 , 48 ( 10 ): 3295 - 3305 .

张天骄 , 李济生 , 李晶 , 等 . 基于混合蚁群优化的天地一体化调度方法 [J ] . 系统工程与电子技术 , 2016 , 38 ( 7 ): 1555 - 1562 .

ZHANG T J , LI J S , LI J , et al . Space-ground integrated scheduling based on the hybrid ant colony optimization [J ] . Systems Engineering and Electronics , 2016 , 38 ( 7 ): 1555 - 1562 .

陶孙杰 , 宋竹 . 一种测控数据传输一体化站网资源调度算法 [J ] . 电讯技术 , 2018 , 58 ( 7 ): 760 - 767 .

TAO S J , SONG Z . A ground-station resource scheduling algorithm based on integration of TT ＆ C and data transmission [J ] . Telecommunication Engineering , 2018 , 58 ( 7 ): 760 - 767 .

金光 , 武小悦 , 高卫斌 . 卫星地面站资源调度优化模型及启发式算法 [J ] . 系统工程与电子技术 , 2004 , 26 ( 12 ): 1839 - 1841 , 1875 .

JIN G , WU X Y , GAO W B . Ground Station resource scheduling optimization model and its heuristic algorithm [J ] . Systems Engineering and Electronics , 2004 , 26 ( 12 ): 1839 - 1841 , 1875 .

金光 , 武小悦 , 高卫斌 . 基于冲突的卫星地面站系统资源调度与能力分析 [J ] . 小型微型计算机系统 , 2007 , 28 ( 2 ): 310 - 312 .

JIN G , WU X Y , GAO W B . Conflict based resource scheduling and capability analysis of satellite-ground station system [J ] . Journal of Chinese Computer Systems , 2007 , 28 ( 2 ): 310 - 312 .

刘嵩 , 白国庆 , 陈英武 . 地球观测网络成像任务可调度性预测方法 [J ] . 宇航学报 , 2015 , 36 ( 5 ): 583 - 588 .

LIU S , BAI G Q , CHEN Y W . Prediction method for imaging task schedulability of earth observation network [J ] . Journal of Astronautics , 2015 , 36 ( 5 ): 583 - 588 .

杜红梅 , 柯宏发 . 基于多智能体技术的航天测控资源调度模型设计 [C ] // 系统仿真技术及其应用 . 2015 ( 16 ): 78 - 81 .

DU H M , KE H F . Design of space TT＆C resource scheduling model based on multi-agent technology [C ] // System simulation technology and its application . 2015 ( 16 ): 78 - 81 .

BADALONI S , FALDA M , GIACOMIN M . Solving temporal over-constrained problems using fuzzy techniques [J ] . Journal of Intelligent ＆ Fuzzy Systems:Applications in Engineering and Technology , 2007 , 18 ( 3 ): 255 - 265 .

WANG H J , YANG Z , ZHOU W G , et al . Online scheduling of image satellites based on neural networks and deep reinforcement learning [J ] . Chinese Journal of Aeronautics , 2019 , 32 ( 4 ): 1011 - 1019 .

ZHANG T J , KE L J , LI J S , et al . Fireworks algorithm for the satellite link scheduling problem in the navigation constellation [C ] // Proceedings of 2016 IEEE Congress on Evolutionary Computation (CEC) . Piscataway:IEEE Press , 2016 : 4029 - 4037 .

窦骄 , 韩孟飞 , 宁金枝 , 等 . 小卫星测控通信技术发展与趋势 [J ] . 航天器工程 , 2021 , 30 ( 6 ): 113 - 119 .

DOU J , HAN M F , NING J Z , et al . Development and trends of TT ＆ C and communication technology for small satellite [J ] . Spacecraft Engineering , 2021 , 30 ( 6 ): 113 - 119 .

陈峰 , 武小悦 . 天地测控资源一体化调度模型 [J ] . 宇航学报 , 2010 , 31 ( 5 ): 1405 - 1412 .

CHEN F , WU X Y . Space and ground TT ＆ C resource integrated scheduling model [J ] . Journal of Astronautics , 2010 , 31 ( 5 ): 1405 - 1412 .

孟欢 . 测控资源调度预测与服务功能链优化映射研究 [D ] . 天津:天津大学 , 2019 .

MENG H . Research on measurement and control resource scheduling prediction and service function chain optimization mapping [D ] . Tianjin:Tianjin University , 2019 .

安元元 . 低轨卫星测控系统调度策略的设计与实现 [D ] . 西安:中国科学院大学(中国科学院国家授时中心) , 2021 .

AN Y Y . Design and realization of schedule strategy of the LEO satellite telemetry and telecontrol system [D ] . Xi'an:National Time Service Center,Chinese Academy of Sciences , 2021 .

雷瀚 , 杨帆 , 刘建平 , 等 . 航天地基测运控资源调度多任务需求分析及统一的任务模型研究 [C ] // 第三届体系工程学术会议论文集——复杂系统与体系工程管理 . 2021 : 100 - 110 .

LEI H , YANG F , LIU J P , et al . Multi-task demand analysis and unified task model research of space-based survey,transportation and control resource scheduling [C ] // Proceedings of the 3rd Symposium on Systems Engineering-Complex Systems and Systems Engineering Management . 2021 : 100 - 110 .

唐成圆 . 面向测运控的卫星网络常规任务规划算法研究 [D ] . 西安:西安电子科技大学 , 2020 .

TANG C Y . Routine task planning algorithm for telemetry tracking and operation control center in satellite network [D ] . Xi'an:Xidian University , 2020 .

武艺 . 基于深度强化学习的多星测控资源调度方法研究 [D ] . 重庆:重庆大学 , 2020 .

WU Y . Research on A scheduling method of TT＆C resources for multi-satellite based on deep reinforcement learning [D ] . Chongqing:Chongqing University , 2020 .

李长德 , 徐伟 , 徐梁 , 等 . 基于深度神经网络的多星测控调度方法 [J ] . 中国空间科学技术 , 2022 , 42 ( 1 ): 65 - 72 .

LI C D , XU W , XU L , et al . Multi-satellite TT＆C scheduling method based on deep neural network [J ] . Chinese Space Science and Technology , 2022 , 42 ( 1 ): 65 - 72 .

LOWE R , WU Y , TAMAR A , et al . Multi-agent actor-critic for mixed cooperative-competitive environments [EB ] . 2017 .

Views

652

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Intelligent Load Balancing Algorithm of Mega Constellation

Related Author

Shuxin LUO

Chao ZHANG

Yong XIAO

Jianping LIU

Related Institution

School of Information and Communication Engineering, Faculty of Electronic and Information Engineering, Xi’an Jiaotong University

State Key Laboratory of Astronautic Dynamics, Xi’an Satellite Control Center

⁰