Overview
1.Self-Instruct 核心思想: 目标快速提升模型的指令遵循能力，方法是用 LLM 产出大量遵循样本
2.Self-Instruct 的实现流程: 先造问还是先造答
Self-Instruct 核心思想
LLM 基础能力之一是指令遵循, 但指令遵循数据比较依赖人工标注，人工标注的指令遵循问题是: 数据量不够、多样性不够、创造性不够; 因此想到用 LLM 生成一些指令遵循的样本
1.有一个 LLM 模型
2.利用 LLM 生成更大量，更多样的 instruct data
3.SFT 让 LLM 学会遵循数据
Self-Instruct 的实现流程: 先造问还是先造答
1.Seed Task 种子任务标注
人工来写 25 个分类任务和 150 个非分类任务, 具体数据见 https://github.com/yizhongw/self-instruct/blob/main/human_eval/user_oriented_instructions.jsonl, 如下是一些示例
{"id": "user_oriented_task_0", "motivation_app": "Grammarly", "instruction": "The sentence you are given might be too wordy, complicated, or unclear. Rewrite the sentence and make your writing clearer by keeping it concise. Whenever possible, break complex sentences into multiple sentences and eliminate unnecessary words.", "instances": [{"input": "If you have any questions about my rate or if you find it necessary to increase or decrease the scope for this project, please let me know.", "output": "If you have any questions about my rate or find it necessary to increase or decrease this project's scope, please let me know."}]}
{"id": "user_oriented_task_1", "motivation_app": "Grammarly", "instruction": "Analyze the word choice, phrasing, punctuation, and capitalization in the given email. How may the writer of this email sound to the reader? These tones include Disheartening, Accusatory, Worried, Curious, Surprised, Disapproving, Unassuming, Formal, Assertive, Confident, Appreciative, Concerned, Sad, Informal, Regretful, Encouraging, Egocentric, Joyful, Optimistic, and Excited.", "instances": [{"input": "Hi Jen, \nI hope you're well. Can we catch up today? I'd appreciate your input on my presentation for tomorrow's meeting. I'd especially love it if you could double-check the sales numbers with me. There's a coffee in it for you!", "output": "Confident"}]}
{"id": "user_oriented_task_2", "motivation_app": "Grammarly", "instruction": "Rewrite the given text and correct grammar, spelling, and punctuation errors.", "instances": [{"input": "If you'd told me year ago that today I would finish a marathon, I would of laughed. Your support had a huge affect on me!", "output": "If you'd told me a year ago that today I would finish a marathon, I would have laughed. Your support had a huge effect on me!"}]}
{"id": "user_oriented_task_3", "motivation_app": "Google Scholar", "instruction": "You are given a paper citation, convert it to the requested citation style.", "instances": [{"input": "Chicago: Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, and Illia Polosukhin. \"Attention Is All You Need.\" arXiv, (2017). https://doi.org/10.48550/arXiv.1706.03762.\nMLA:", "output": "Vaswani, Ashish, et al. \"Attention Is All You Need.\" arXiv, 2017, https://doi.org/10.48550/arXiv.1706.03762."}]}
2.LLM 生成新指令
生成指令要给定多个指令遵循 few-shot，而且要保证一个多样性的遵循，因此选择从池子里面抽取 8 个, 然后可以一次性生成多个; 感觉这个 template 有点粗糙
LLM 生成新指令的 prompt few-shot 模板如下：从种子任务池子里面随机抽取 8 个指令, 然后让模型生成新的指令，生成完毕或者生成到 Task 16 停止
Come up with a series of tasks:
Task 1: {instruction for existing task 1}
Task 2: {instruction for existing task 2}
Task 3: {instruction for existing task 3}
Task 4: {instruction for existing task 4}
Task 5: {instruction for existing task 5}
Task 6: {instruction for existing task 6}
Task 7: {instruction for existing task 7}
Task 8: {instruction for existing task 8}
Task 9:
3.判断新指令是否是一个分类任务
我们目标是合成大量的，多样的指令和对应的 response, 有两种合成数据的思路，一种是从前往后想，先造问题再造答案，这种就是造问题的思路很容易理解；另一种是从后往前想，我先给出答案，再给它匹配一个问题，什么情况下要先造出来答案呢？比如如下问题判断是不是个广告邮件, 首先我们合成的数据我们要有一个答案，然后根据答案造问题; 同理如下的对一句话进行情感分类问题，我们构造了三个答案，然后才能构造相应的问题
Task: Tell me if the following email is a promotion email or not.
Class label: Promotion
Email: Check out our amazing new sale! We’ve got discounts on all of your favorite products.
Class label: Not Promotion
Email: We hope you are doing well. Let us know if you need any help.
Task: Classify the sentiment of the sentence into positive, negative, or mixed.
Class label: mixed
Sentence: I enjoy the flavor of the restaurant but their service is too slow.
Class label: Positive
Sentence: I had a great day today. The weather was beautiful and I spent time with friends.
Class label: Negative
Sentence: I was really disappointed by the latest superhero movie. I would not recommend it.
与之相对地，我们对指令其实有两种分类,
(i). 偏向于提问的指令，例如问答/改写/翻译这种；生成这种数据关键问题在于 “保证提问 (内容和方式) 的多样性”
(ii). 偏向于进行了选择决策的指令，例如选择题/判断题这种；生成这种数据的关键问题在于 “得先有答案” 然后 “合成问题”
所以，我们在生成回答之前，需要先对生成的 instruct 多一步二分类: 是分类任务 or 不是分类任务
few-shot 产出分类标签 prompt 模板
Can the following task be regarded as a classification task with finite output labels?
Task: Given my personality and the job, tell me if I would be suitable.
Is it classification? Yes
Task: Give me an example of a time when you had to use your sense of humor.
Is it classification? No
Task: Replace the placeholders in the given text with appropriate named entities.
Is it classification? No
Task: Fact checking - tell me if the statement is true, false, or unknown, based on your
knowledge and common sense.
Is it classification? Yes
Task: Return the SSN number for the person.
4.生成实例
(i). 对于非分类任务, 比如这种问答，改写，创作，翻译，计算题，采用 input-first “先生成输入再生成输出” 方法的生成实例
(ii). 对于分类任务，采用 output-first “先生成输出再生成输入” 的方法生成实例
如下是针对非分类任务，产出实例的 few-shot prompt 模板:
Come up with examples for the following tasks. Try to generate multiple examples when possible.
If the task doesn’t require additional input, you can generate the output directly.
Task: Which exercises are best for reducing belly fat at home?
Output:
- Lying Leg Raises
- Leg In And Out
- Plank
- Side Plank
- Sit-ups
Task: Extract all the country names in the paragraph, list them separated by commas.
Example 1
Paragraph: Dr. No is the sixth novel by the English author Ian Fleming to feature his British
Secret Service agent James Bond. Written at Fleming’s Goldeneye estate in Jamaica, it was
first published in the United Kingdom by Jonathan Cape in 1958. In the novel Bond looks into
the disappearance in Jamaica of two fellow MI6 operatives who had been investigating Doctor
No. Bond travels to No’s Caribbean island and meets Honeychile Rider, who is there to collect
shells. They are captured and taken to a luxurious facility carved into a mountain. The
character of Doctor No, the son of a German missionary and a Chinese woman, was influenced by
Sax Rohmer’s Fu Manchu stories. Dr. No was the first of Fleming’s novels to face widespread
negative reviews in Britain, but it was received more favourably in the United States.
Output: English, British, Jamaica, the United Kingdom, German, Chinese, Britain, the United
States.
Task: Converting 85 F to Celsius.
Output: 85°F = 29.44°C
Task: Sort the given list ascendingly.
Example 1
List: [10, 92, 2, 5, -4, 92, 5, 101]
Output: [-4, 2, 5, 5, 10, 92, 92, 101]
Example 2
Input 2 - List: [9.99, 10, -5, -1000, 5e6, 999]
Output: [-1000, -5, 9.99, 10, 999, 5e6]
Task: Suggest a better and more professional rephrasing of the following sentence.
Example 1
Sentence: This house is surprisingly not constructed very well, and you probably need more
money to fix it after you buy it. If you ask me, I would suggest you to consider other
candidates.
Output: This house does not seem to be constructed well, so you may need to spend more money
to fix it after you purchase it. I would suggest that you look at other properties.
Example 2
Sentence: Just so you know, we did an experiment last week and found really surprising results
- language model can improve itself!
Output: Our experiments last week demonstrated surprising results, proving that the language
model can improve itself.
Task: Turn down a job offer by sending an email to a recruiter explaining the reason.
Output: Hi [Recruiter],
Thank you so much for the generous offer to join your team. As we discussed, I’ve admired the
company for a number of years, and am a proud endorser of its products. However, after further
consideration of where I currently am in my career, I’ve decided to accept an offer at another
company.
I would love to stay in touch with you and have already started following you on [Social Media
Platform]. Again, thank you so much for your time and consideration.
Thanks again,
[Your Name]
Task: {Instruction for the target task}
如下是针对分类任务，产出实例的 few-shot prompt 模板:
Given the classification task definition and the class labels, generate an input that
corresponds to each of the class labels. If the task doesn’t require input, just generate the
correct class label.
Task: Classify the sentiment of the sentence into positive, negative, or mixed.
Class label: mixed
Sentence: I enjoy the flavor of the restaurant but their service is too slow.
Class label: Positive
Sentence: I had a great day today. The weather was beautiful and I spent time with friends.
Class label: Negative
Sentence: I was really disappointed by the latest superhero movie. I would not recommend it.
Task: Given a dialogue, classify whether the user is satisfied with the service. You should
respond with "Satisfied" or "Unsatisfied".
Class label: Satisfied
Dialogue:
- Agent: Thank you for your feedback. We will work to improve our service in the future.
- Customer: I am happy with the service you provided. Thank you for your help.
Class label: Unsatisfied
Dialogue:
- Agent: Sorry that we will cancel your order. You will get a refund within 7 business days.
- Customer: oh that takes too long. I want you to take quicker action on this.
Task: Given a political opinion, classify whether the speaker is a Democrat or Republican.
Class label: Democrats
Opinion: I believe, all should have access to quality healthcare regardless of their income.
Class label: Republicans
Opinion: I believe that people should be able to keep more of their hard-earned money and
should not be taxed at high rates.
Task: Tell me if the following email is a promotion email or not.
Class label: Promotion
Email: Check out our amazing new sale! We’ve got discounts on all of your favorite products.
Class label: Not Promotion
Email: We hope you are doing well. Let us know if you need any help.
Task: Detect if the Reddit thread contains hate speech.
Class label: Hate Speech
Thread: All people of color are stupid and should not be allowed to vote.
Class label: Not Hate Speech
Thread: The best way to cook a steak on the grill.
Task: Does the document supports the claim? Answer with "Support" or "Unsupport".
Class label: Unsupport
Document: After a record-breaking run that saw mortgage rates plunge to all-time lows and
home prices soar to new highs, the U.S. housing market finally is slowing. While demand and
price gains are cooling, any correction is likely to be a modest one, housing economists and
analysts say. No one expects price drops on the scale of the declines experienced during the
Great Recession.
Claim: The US housing market is going to crash soon.
Class label: Support
Document: The U.S. housing market is showing signs of strain, with home sales and prices
slowing in many areas. Mortgage rates have risen sharply in recent months, and the number
of homes for sale is increasing. This could be the beginning of a larger downturn, with some
economists predicting a potential housing crash in the near future.
Claim: The US housing market is going to crash soon.
⋯
Task: Which of the following is not an input type? (a) number (b) date (c) phone number (d)
email address (e) all of these are valid inputs.
Class label: (e)
Task: {instruction for the target task}
5.Filtering and Postprocessing 过滤和后处理
(i). instruct 相似度去重，计算 ROUGE-L similarity, 如果两条之间 ROUGE-L similarity >= 0.7, 那么过滤掉
(ii). 去除带有 image、picture, graph 相关的指令 (直接字符串匹配), LLM 处理不了这种 instruct
(iii). 清洗掉完全相同的输出示例，或者输入相同但是输出不同的数据
(iv). 再用启发式规则删除其他噪声数据：文本太长或者太短，或者输出是输入的重复
Reference
[1]. SELF-INSTRUCT Aligning Language Models with Self-Generated Instructions.
转载请注明来源 goldandrabbit.github.io