Search Results for author:
Found 11 papers, 5 papers with code
Reward Generalization in RLHF: A Topological Perspective
As a solution, we introduce a theoretical framework for investigating reward generalization in reinforcement learning from human feedback (RLHF), focusing on the topology of information flow at both macro and micro levels.
Aligner: Efficient Alignment by Learning to Correct
However, the tension between the complexity of current alignment methods and the need for rapid iteration in deployment scenarios necessitates the development of a model-agnostic alignment approach that can operate under these constraints.
AI Alignment: A Comprehensive Survey
The former aims to make AI systems aligned via alignment training, while the latter aims to gain evidence about the systems' alignment and govern them appropriately to avoid exacerbating misalignment risks.
Safe RLHF: Safe Reinforcement Learning from Human Feedback
However, the inherent tension between the objectives of helpfulness and harmlessness presents a significant challenge during LLM training.
Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark
By introducing this benchmark, we aim to facilitate the evaluation and comparison of safety performance, thus fostering the development of reinforcement learning for safer, more reliable, and responsible real-world applications.
Evolving Diverse Red-team Language Models in Multi-round Multi-agent Games
Furthermore, we develop a Gamified Red Team Solver (GRTS) with diversity measures to mitigate mode collapse and theoretically guarantee the convergence of approximate Nash equilibrium which results in better strategies for both teams.
Baichuan 2: Open Large-scale Language Models
Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering.
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
In this paper, we introduce the BeaverTails dataset, aimed at fostering research on safety alignment in large language models (LLMs).
OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research
AI systems empowered by reinforcement learning (RL) algorithms harbor the immense potential to catalyze societal advancement, yet their deployment is often impeded by significant safety concerns.
Proactive Multi-Camera Collaboration For 3D Human Pose Estimation
This paper presents a multi-agent reinforcement learning (MARL) scheme for proactive Multi-Camera Collaboration in 3D Human Pose Estimation in dynamic human crowds.
TorchOpt: An Efficient Library for Differentiable Optimization
TorchOpt further provides a high-performance distributed execution runtime.
