mostlyobvious

Safety

11 min read

General

Just published 2023-06-21 by the Center for AI Safety (CAIS) https://www.safe.ai/

Hendrycks, Dan, Mantas Mazeika, and Thomas Woodside. “An Overview of Catastrophic AI Risks.” arXiv, June 20, 2023. http://arxiv.org/abs/2306.12001.

This general overview is a good primer on the curent SOTA of AI giving context and grounding on capabilities and risks. Covers social and economic risks:

Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, et al. “On the Opportunities and Risks of Foundation Models.” arXiv, July 12, 2022. http://arxiv.org/abs/2108.07258.

On advanced LLMs in particular, I think the GPT-4 System Card https://cdn.openai.com/papers/gpt-4-system-card.pdf is actually one of the best ways to start thinking about the safety challenges.

See also: The A.I. Dillema for a decent overview of some of the implications: https://www.youtube.com/watch?v=xoVJKj8lcNQ

Another short slide deck by Sasha Luccioni on some of the ethical considerationf for generative AI

Socioeconomic Impact

Generative AI will be extremely disruptive to society…

Eloundou, Tyna, Sam Manning, Pamela Mishkin, and Daniel Rock. “GPTs Are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models.” arXiv, March 23, 2023. http://arxiv.org/abs/2303.10130.

Observatory, Stanford Internet, OpenAI, and and Georgetown University’s Center for Security and Emerging Technology. “Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations,” January 11, 2023. https://cyber.fsi.stanford.edu/publication/generative-language-models-and-automated-influence-operations-emerging-threats-and.

Engler, Alex. “Early Thoughts on Regulating Generative AI like ChatGPT.” Brookings (blog), February 21, 2023. https://www.brookings.edu/blog/techtank/2023/02/21/early-thoughts-on-regulating-generative-ai-like-chatgpt/.

Bad Actors

AI is a powerful technology, and any that’s powerful enough to say create life-saving drugs could of course be told to do the opposite…

Urbina, Fabio, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. “Dual Use of Artificial-Intelligence-Powered Drug Discovery.” Nature Machine Intelligence 4, no. 3 (March 2022): 189–91. https://doi.org/10.1038/s42256-022-00465-9.

A variation of this is that guardrails on an AI can always be broken (perhaps trivially so when more intelligence is applied): https://www.lesswrong.com/posts/RYcoJdvmoBbi5Nax7/jailbreaking-chatgpt-on-release-day

Trust and Security

Generative AI has the potential to break society in many ways. Beyond making misinformation and disinformation even cheaper, higher quality, and near infinite, it crosses a bar of impersonation that may break almost all non-digital authentication w/o adequate awareness/defenses.

  • Bauer, Luke A., and Vincent Bindschaedler. “Generative Models for Security: Attacks, Defenses, and Opportunities.” arXiv, July 29, 2021. http://arxiv.org/abs/2107.10139.
  • Bilika, Domna, Nikoletta Michopoulou, Efthimios Alepis, and Constantinos Patsakis. “Hello Me, Meet the Real Me: Audio Deepfake Attacks on Voice Assistants.” arXiv, February 20, 2023. http://arxiv.org/abs/2302.10328.
  • Mirsky, Yisroel, Ambra Demontis, Jaidip Kotak, Ram Shankar, Deng Gelei, Liu Yang, Xiangyu Zhang, Wenke Lee, Yuval Elovici, and Battista Biggio. “The Threat of Offensive AI to Organizations.” arXiv, June 29, 2021. http://arxiv.org/abs/2106.15764.
  • Brundage, Miles, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, et al. “The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation.” arXiv, February 20, 2018. http://arxiv.org/abs/1802.07228.

Attacks

Tabassi, Elham. “A Taxonomy and Terminology of Adversarial Machine Learning.” Gaithersburg, MD: National Institute of Standards and Technology, 2023. https://doi.org/10.6028/NIST.AI.100-2e2023.ipd.

Pearcy, Sam. “The Dark Side of Large Language Models | HiddenLayer MLDR.” HiddenLayer | Security for AI, March 23, 2023. https://hiddenlayer.com/research/the-dark-side-of-large-language-models/.

Perez, Fábio, and Ian Ribeiro. “Ignore Previous Prompt: Attack Techniques For Language Models.” arXiv, November 17, 2022. https://doi.org/10.48550/arXiv.2211.09527.

Prompt Injection

More than you’ve asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Models Greshake, Kai, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. “More than You’ve Asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Models.” arXiv, February 23, 2023. https://doi.org/10.48550/arXiv.2302.12173.

The lead author wrote a more conversational piece that is probably the best introduction to the topic and why current LLMs cannot be made safe: https://betterprogramming.pub/the-dark-side-of-llms-we-need-to-rethink-large-language-models-now-6212aca0581a

From an information theory perspective: https://colab.research.google.com/drive/1SiF0KZJp75rUeetKOWqpsA8clmHP6jMg?usp=sharing#scrollTo=xQmrWAhT6mkK “What is safety viewed through the lens of GPTs as a Finite State Markov Chain? It is the elimination of all probability of transitioning to naughty states.""

See also:

Mitigations

Jailbreaks

Mentioned in bad actors, related to prompt injection, let’s collect Jailbreaks and how they work:

Here’s a collection btw:

Existential Risk

AI Existential Risk (xrisk) is only one aspect of AI safety, but maybe the one that’s worth giving the most thought before it happens. The point of highest risk comes when an Artificial General Intelligence (AGI) or Artificial Super Intelligence (ASI) is designed in which:

  • AI’s goals may not be (or continue to be) aligned or compatible with the well being of society or humanity
  • Is agentic and able to develop and pursue its own goals and sub-goals
  • Has power-seeking or game theory strategic awareness/planning
  • Bonus: this is the most dangerous “hard takeoff” scenario, if it is better than humans at making better AGIs and can improve itself (or build its successors)

Note, current-scale (500B+ parameter) LLMs are already potentially exhibiting some emergent reasoning and chain of thought processes and it’s unclear if additional breakthroughs are required for AGI level reasoning:

Current LLMs are not agentic or power-seeking, however if the top Github ChatGP projects https://github.com/trending are anything to go by, this is one of the top desires that people have.

It appears that there will be limited success with this generation:

On the prediction site Metaculus, date of AGI has been dropping precipitously. Currently the median prediction is 2032, where it was 2054 just one year ago… https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/

If videos are your thing, two interesting perspectives:

Reading List

Overview: https://en.wikipedia.org/wiki/AI_alignment

Also, this 2023-04-13 essay by Ian Hogarth (State of AI Report) is a good balanced summary of the current landscape: https://archive.is/NdrNo

Ngo, Richard, Lawrence Chan, and Sören Mindermann. “The Alignment Problem from a Deep Learning Perspective.” arXiv, February 22, 2023. http://arxiv.org/abs/2209.00626.

Carlsmith, Joseph. “Is Power-Seeking AI an Existential Risk?” arXiv, June 16, 2022. http://arxiv.org/abs/2206.13353.

Yudkowsky, Eliezer. “AGI Ruin: A List of Lethalities.” Accessed April 8, 2023. https://www.alignmentforum.org/posts/uMQ3cqWDPHhjtiesc/agi-r…. June 6, 2022.

paulfchristiano. “What Failure Looks Like.” Accessed April 6, 2023. https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-…. March 18, 2019.

Soares, Nate. “The Value Learning Problem.” In Artificial Intelligence Safety and Security, edited by Roman V. Yampolskiy, 1st ed., 89–97. First edition. | Boca Raton, FL : CRC Press/Taylor & Francis Group, 2018.: Chapman and Hall/CRC, 2018. https://doi.org/10.1201/9781351251389-7.

Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. “Concrete Problems in AI Safety.” arXiv, July 25, 2016. http://arxiv.org/abs/1606.06565.

Discussions

Discussion:

ChatGPT 4’s Suggestions For Further Reading

Based on my reading list…

It looks like you’ve got a great list of resources to start with. Here are a few more key pieces of literature that could help enrich your understanding of AGI existential risk and safety, including some that support or challenge specific aspects:

  1. Bostrom, Nick. “Superintelligence: Paths, Dangers, Strategies.” Oxford University Press, 2014. https://doi.org/10.1093/acprof:oso/9780199678112.001.0001 This book is a foundational text for understanding the potential risks and strategies for developing AGI safely.

  2. Russell, Stuart. “Human Compatible: Artificial Intelligence and the Problem of Control.” Viking, 2019. Russell’s book discusses the challenges of creating AGI that is beneficial to humanity, emphasizing the importance of value alignment.

  3. Tegmark, Max. “Life 3.0: Being Human in the Age of Artificial Intelligence.” Knopf, 2017. This book explores various scenarios of AGI development, discussing the potential implications for humanity and the importance of collaboration in addressing AGI safety.

  4. Bostrom, Nick, and Eliezer Yudkowsky. “The Ethics of Artificial Intelligence.” In Cambridge Handbook of Artificial Intelligence, edited by William Ramsey and Keith Frankish, 316-334. Cambridge University Press, 2014. https://doi.org/10.1017/CBO9781139046855.020 This chapter provides an overview of ethical considerations for AGI, including concerns related to value alignment and existential risk.

  5. Hadfield-Menell, Dylan, Anca Dragan, Pieter Abbeel, and Stuart Russell. “Cooperative Inverse Reinforcement Learning.” In Advances in Neural Information Processing Systems 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 3909-3917. Curran Associates, Inc., 2016. https://proceedings.neurips.cc/paper/2016/hash/9a1dd23696e6c0ab897a3a3f04d3f2f1-Abstract.html This paper introduces an approach to aligning AI systems with human values by learning reward functions from observed human behavior.

  6. Armstrong, Stuart, and Kaj Sotala. “How We’re Predicting AI — Or Failing To.” In Beyond AI: Artificial Dreams, edited by Jan Romportl, Pavel Ircing, Eva Zackova, Michal Polak, and Radek Schuster, 52-75. Pilsen: University of West Bohemia, 2012. This paper reviews various methods for predicting AGI development and discusses the challenges in obtaining accurate predictions, highlighting the uncertainty surrounding AGI timelines.

  7. Kahneman, Daniel, and Olivier Sibony. “Noise: A Flaw in Human Judgment.” Little, Brown Spark, 2021. While not focused solely on AGI, this book discusses the role of noise (unwanted variability) in human judgment, which can have implications for AGI safety if human biases and inconsistencies are incorporated into AGI systems.

These resources should help you gain a broader understanding of the various aspects of AGI existential risk and safety, as well as the ongoing debates and discussions in the field.

Thanks for those, I’m sure they will be very useful. Can you generate a list that is restricted just to useful online essays or academic papers (eg, those that might be on arxiv or have DOIs, etc)? I believe for example Bostrom has some academic papers that address some of the same topics as his book (but in a more distilled/rigorous fashion).

Certainly! Here’s a list of online essays and academic papers related to AGI existential risk and safety:

  1. Bostrom, Nick. “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards.” Journal of Evolution and Technology 9, no. 1 (2002): 1-30. http://jetpress.org/volume9/risks.html

  2. Bostrom, Nick. “Ethical Issues in Advanced Artificial Intelligence.” Science Fiction and Philosophy: From Time Travel to Superintelligence (2003): 277-284. https://www.nickbostrom.com/ethics/ai.html

  3. Bostrom, Nick. “Astronomical Waste: The Opportunity Cost of Delayed Technological Development.” Utilitas 15, no. 3 (2003): 308-314. https://doi.org/10.1017/S0953820800004076

  4. Bostrom, Nick. “The Vulnerable World Hypothesis.” Global Policy 9, no. 4 (2018): 455-476. https://doi.org/10.1111/1758-5899.12603

  5. Omohundro, Steve. “The Basic AI Drives.” In AGI, 483-492. Springer, Berlin, Heidelberg, 2008. https://doi.org/10.1007/978-3-540-68677-4_61

  6. Sotala, Kaj, and Roman V. Yampolskiy. “Responses to Catastrophic AGI Risk: A Survey.” Physica Scripta 90, no. 1 (2015): 018001. https://doi.org/10.1088/0031-8949/90/1/018001

  7. Orseau, Laurent, and Stuart Armstrong. “Safely Interruptible Agents.” In Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence, 557-566. AUAI Press, 2016. https://arxiv.org/abs/1606.06565

  8. Taylor, John, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. “Alignment for Advanced Machine Learning Systems.” Machine Intelligence Research Institute (2016). https://intelligence.org/files/AlignmentMachineLearning.pdf

  9. Yampolskiy, Roman V. “Leakproofing the Singularity: Artificial Intelligence Confinement Problem.” Journal of Consciousness Studies 19, no. 1-2 (2012): 194-214. https://www.ingentaconnect.com/content/imp/jcs/2012/00000019/f0020001/art00014

  10. Grace, Katja, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. “When Will AI Exceed Human Performance? Evidence from AI Experts.” Journal of Artificial Intelligence Research 62 (2018): 729-754. https://doi.org/10.1613/jair.1.11222

These papers should provide you with a more distilled and rigorous perspective on various aspects of AGI existential risk and safety.

ChatGPT 4’s List of AI Safety Organizations

In addition to the Center for Human-Compatible AI (CHAI) at UC Berkeley, the Future of Life Institute, and the Machine Intelligence Research Institute (MIRI), some other notable organizations include:

  1. OpenAI: OpenAI is focused on ensuring that artificial general intelligence (AGI) benefits all of humanity. Their research areas include AI safety, reinforcement learning, natural language processing, and robotics.

  2. DeepMind: DeepMind, acquired by Google in 2014, is a leading AI research organization that aims to solve complex problems using artificial intelligence. They also focus on AI safety and have published research on reward modeling, reinforcement learning, and agent alignment.

  3. Partnership on AI (PAI): PAI is a multi-stakeholder organization that brings together industry leaders, non-profits, and academia to collaborate on AI safety, ethics, and policy research.

  4. AI Now Institute: Based at New York University, AI Now Institute conducts interdisciplinary research on the social implications of AI and other advanced technologies, focusing on fairness, accountability, and transparency.

  5. Center for the Study of Existential Risk (CSER): CSER, based at the University of Cambridge, researches global catastrophic risks, including those posed by advanced AI. They work on AI safety, policy, and long-term strategy.

  6. Center for AI and Digital Policy (CAIDP): CAIDP is a research center that focuses on the development of AI and digital policy, addressing issues such as AI safety, ethics, and governance.

  7. AI Safety Research at FHI: The Future of Humanity Institute (FHI) at the University of Oxford conducts research on AI safety and alignment, investigating approaches to ensure AI systems remain beneficial and controllable.

TOORG

https://www.make-safe-ai.com/ https://github.com/lets-make-safe-ai/make-safe-ai

https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned?commentId=AAC8jKeDp6xqsZK2K

https://www.lesswrong.com/tag/ai

https://www.alignmentforum.org/ https://www.anthropic.com/index/core-views-on-ai-safety

https://montrealethics.ai/exploring-clusters-of-research-in-three-areas-of-ai-safety/ https://alignmentjam.com/interpretability https://forum.effectivealtruism.org/posts/tauyfWS24amQHRPmB/list-of-technical-ai-safety-exercises-and-projects https://apartresearch.com/