On September 26, the papers for NeurIPS 2024 (Annual Conference on Neural Information Processing Systems) were announced. NeurIPS is one of the most prestigious academic conferences in artificial intelligence globally, ranked 7th in the influence of global academic journals and conferences published by Google Scholar, with an H5 index of 337. It is considered one of the “three major conferences” in AI, along with ICML and ICLR, known for their high difficulty, top-level quality, and strong influence. According to official data, this year’s conference received 15,671 valid paper submissions, with an acceptance rate of only 25.8%. The paper from the team of Department of Electrical Engineering and Applied Electronics (EEA), Tsinghua University, “GLinSAT: The General Linear Satisfiability Neural Network Layer By Accelerated Gradient Descent,” received acceptance notification, making it the first paper from EEA to be included in the main conference of NeurIPS. The first author is EEA PhD student Zeng Hongtai, with corresponding author being EEA Professor Guo Qinglai. The author team also includes EEA Dr. Zhou Yanzhen, as well as Yang Chao and Yang Cheng from Alibaba DAMO Academy. The research was supported by funding from the National Natural Science Foundation’s joint fund with the Southern Power Grid for “Knowledge Discovery Research on New Type Power System Operation Modes Based on Hybrid Intelligence,” the National Natural Science Foundation’s Innovative Research Group Project for “Safety Analysis and Intelligent Control of New Type Power Systems,” and the AIR project from Alibaba DAMO Academy for “AI-Based Research on Large-Scale Real-Time Scheduling Enhancement Technology for New Type Power Systems.”

Paper Overview

In recent years, with the significant success of neural networks in regression tasks, many researchers have attempted to use them to solve complex constrained optimization (decision) problems. However, ensuring that the output of neural networks meets specified constraints remains a critical issue that needs to be addressed. Standard activation functions (like ReLU, Sigmoid, and Softmax) can impose some simple constraints but struggle with more general linear constraints. A common approach to ensure that neural network outputs meet constraints is to introduce penalty terms in the loss function; however, this method faces challenges in selecting penalty coefficients and theoretically bounding constraint violations.

To address this, the team proposed GLinSAT, a neural network activation layer designed to satisfy general linear constraints. They modeled the satisfiability problem of general linear constraints as an entropy-regularized optimization problem with linear constraints and demonstrated that this problem can be transformed into an unconstrained convex optimization problem with Lipschitz continuous gradients. Building on this, the team developed a differentiable solver that does not require matrix decomposition to address the above problems, utilizing accelerated gradient descent and the parallelism of GPUs to enhance both forward and backward propagation. Compared to existing constraint-satisfaction layers, this method is the first activation layer for neural networks to satisfy general linear constraints, where all operations are differentiable and do not require matrix decomposition, significantly improving training efficiency while ensuring constraint satisfaction. Through experiments on constrained traveling salesman problems, partially matched graphs with outliers, predictive portfolio allocation, and unit commitment problems in power systems, the team demonstrated that GLinSAT can achieve tenfold or even hundredfold acceleration compared to existing constraint satisfaction layers.

Workflow of GLinSAT