趋近智
趋近智
改进合成数据集需要有效的数据筛选和清洗流程。生成大量合成数据仅仅是第一步；确保其质量、相关性和安全性对于大型语言模型（LLM）的预训练 (pre-training)和微调 (fine-tuning)来说不可或缺。本练习将引导你构建一个Python脚本来筛选合成文本数据集，这是任何自动化数据质量保证流程的主要构成部分。
我们将重心放在构建一个灵活的脚本，它能根据长度、重复性、占位符文本的存在以及简单的内容启发式等常见标准，识别并移除不理想的数据点。这次动手实践将使你具备能力，开始构建针对你的具体数据和模型要求量身定制的筛选流程。
合成数据生成虽然功能强大，但并非总是完美无缺。生成的数据样本可能过短、过于冗长、包含重复短语、遗留模板标记 (token)，甚至呈现出不理想的内容模式。将此类噪声数据输入到大型语言模型中，可能会降低其性能，引入偏见，或导致模型行为不可预测。一个精心设计的筛选脚本充当质量关卡，确保只有高价值数据进入训练阶段。
假设我们有一个由合成生成的指令-响应对组成的数据集，这可能是使用前面讨论过的大型语言模型方法之一创建的。我们的目标是清洗这个数据集。
我们的脚本将由几个部分组成：
True，否则返回False。以下是说明数据通过我们筛选脚本的一般流程图。
该图显示了原始合成数据如何加载并按顺序通过各种筛选器。未能通过任何筛选器的数据被移动到“已丢弃数据”集合中，而通过所有筛选器的数据则构成“已清洗合成数据集”。
我们开始构建脚本。首先，确保你已安装Python。你可能还需要安装一个用于文本复杂性度量的库，例如textstat。你可以使用pip安装它：
pip install textstat
现在，让我们创建Python脚本，filter_synthetic_data.py。
我们需要json来处理JSONL数据，re用于正则表达式（对占位符检测很有用），collections.Counter用于重复性检测中的n-gram分析，以及textstat用于复杂性分数计算。
import json
import re
from collections import Counter
import textstat # 用于Flesch阅读易度分数
# 筛选器的配置
MIN_RESPONSE_WORDS = 10
MAX_RESPONSE_WORDS = 300
MAX_TRIGRAM_REPETITION_RATIO = 0.3 # 最多30%的三元组可以是出现频率最高的那个
PLACEHOLDER_PATTERNS = [
r"\[insert.*here\]", r"\(your answer\)", r"__+", r"YOUR_RESPONSE_HERE"
]
FORBIDDEN_KEYWORDS = ["unsafe_content", "problematic_phrase"] # 示例关键词
MIN_FLESCH_READING_EASE = 30.0 # 分数低于30被认为难以阅读（更复杂）
# 词数辅助函数
def word_count(text):
return len(text.split())
每个筛选函数都将接收相关文本（例如，模型的响应），如果通过则返回True，如果失败则同时返回原因。
此筛选器检查响应长度（以词为单位）是否落在指定范围内。
def filter_by_length(text, min_words, max_words):
count = word_count(text)
if not (min_words <= count <= max_words):
return False, f"长度 ({count} 词) 超出范围 [{min_words}, {max_words}]"
return True, ""
为了检测过度重复，我们可以查看n-gram频率。这里，我们将检查三元组（trigram）的重复。如果单个三元组在所有三元组中占了过大的比例，则文本可能存在重复。
def get_ngrams(text, n):
words = text.lower().split()
return [" ".join(words[i:i+n]) for i in range(len(words)-n+1)]
def filter_by_repetition(text, n=3, max_ratio=0.3):
if not text.strip(): # 避免空字符串的除零错误
return True, "" # 或者 False, "空文本"，取决于期望行为
ngrams = get_ngrams(text, n)
if not ngrams: # 词数不足以生成n-gram
return True, "" # 如果文本过短不足以进行n-gram分析，则通过（长度筛选器会处理短文本）
counts = Counter(ngrams)
most_common_ngram_count = counts.most_common(1)[0][1]
repetition_ratio = most_common_ngram_count / len(ngrams)
if repetition_ratio > max_ratio:
return False, f"{n}-gram重复率过高: {repetition_ratio:.2f} > {max_ratio}"
return True, ""
此筛选器使用正则表达式来查找常见的占位符模式。
def filter_by_placeholder(text, patterns):
for pattern in patterns:
if re.search(pattern, text, re.IGNORECASE):
return False, f"发现占位符：匹配到 '{pattern}'"
return True, ""
一个简单的筛选器，用于检查是否存在特定的不良关键词。实际应用中，这会更完善，可能使用专门的内容审核API或模型。
def filter_by_keyword(text, keywords):
text_lower = text.lower()
for keyword in keywords:
if keyword.lower() in text_lower:
return False, f"发现禁用关键词：'{keyword}'"
return True, ""
此筛选器使用Flesch阅读易度分数。分数越高表明可读性越强。根据具体用例，我们可能希望筛选掉过于简单或过于复杂的响应。在本例中，如果文本分数高于某个阈值，我们会筛选掉过于简单的文本；如果分数低于另一个阈值，则筛选掉过于复杂的文本。本例中，我们将确保文本的最低复杂程度。
def filter_by_complexity_flesch(text, min_score):
# textstat需要至少100个词才能计算某些统计数据，但Flesch也适用于较短的文本。
# 对于较长的文本，它通常更可靠。
# 如果需要，我们可以在这里添加一个词数检查。
if word_count(text) < 5: # Flesch分数对于非常短的文本可能不可靠
return True, "文本过短无法获得可靠的复杂性分数，通过。"
try:
score = textstat.flesch_reading_ease(text)
if score < min_score: # 分数越低意味着文本越复杂
return False, f"Flesch阅读易度分数 ({score:.2f}) 低于最低值 ({min_score})，过于复杂。"
# 示例：也可以筛选过于简单的文本：
# if score > max_score:
# return False, f"Flesch阅读易度分数 ({score:.2f}) 高于最大值 ({max_score})，过于简单。"
except Exception as e:
# textstat可能在某些极端输入情况下失败
return True, f"无法计算Flesch分数：{e}，通过。"
return True, ""
注意：Flesch阅读易度分数通常范围在0到100之间。分数在60-70之间被认为是普通英语水平。0-30分通常最适合大学毕业生阅读（更复杂）。我们的MIN_FLESCH_READING_EASE = 30.0意味着我们希望丢弃非常复杂的文本。如果你想丢弃过于简单的文本，请调整逻辑。对于本例，让我们调整思路：我们希望确保文本不过于困难。因此，分数越低意味着越困难。我们希望分数高于某个最小值。让我们调整筛选器的逻辑和参数 (parameter)名称，以反映筛选掉过于复杂的文本。
对MIN_FLESCH_READING_EASE的更正后的理解是：我们希望文本至少达到这个分数（即，不太难）。因此，如果score < min_score，则文本过于困难（复杂），我们会将其筛选掉。最初的MIN_FLESCH_READING_EASE = 30.0意味着我们筛选掉比大学毕业生水平更难的文本。这似乎是一个合理的起点。
现在，让我们编写脚本的主要部分，它负责加载数据、应用这些筛选器并保存结果。我们假设输入数据位于JSONL文件中，其中每一行都是一个JSON对象，如{"instruction": "...", "response": "..."}。
def process_dataset(input_filepath, output_filepath_passed, output_filepath_failed):
passed_items = []
failed_items_log = []
filters_to_apply = [
("length", lambda item: filter_by_length(item['response'], MIN_RESPONSE_WORDS, MAX_RESPONSE_WORDS)),
("repetition", lambda item: filter_by_repetition(item['response'], max_ratio=MAX_TRIGRAM_REPETITION_RATIO)),
("placeholder", lambda item: filter_by_placeholder(item['response'], PLACEHOLDER_PATTERNS)),
("keyword", lambda item: filter_by_keyword(item['response'], FORBIDDEN_KEYWORDS)),
("complexity", lambda item: filter_by_complexity_flesch(item['response'], MIN_FLESCH_READING_EASE))
]
try:
with open(input_filepath, 'r', encoding='utf-8') as infile:
for i, line in enumerate(infile):
try:
item = json.loads(line)
if 'response' not in item: # 基本验证
failed_items_log.append({"item_index": i, "original_item": item, "reason": "缺少 'response' 字段"})
continue
original_item_for_log = item.copy() # 如果失败，保留原始项用于记录
passes_all_filters = True
fail_reason = ""
for filter_name, filter_func in filters_to_apply:
passed, reason = filter_func(item)
if not passed:
passes_all_filters = False
fail_reason = f"{filter_name} 筛选失败: {reason}"
break # 在第一次失败时停止
if passes_all_filters:
passed_items.append(item)
else:
failed_items_log.append({"item_index": i, "original_item": original_item_for_log, "reason": fail_reason})
except json.JSONDecodeError:
failed_items_log.append({"item_index": i, "line_content": line.strip(), "reason": "JSON解码错误"})
continue
except FileNotFoundError:
print(f"错误：输入文件未在 {input_filepath} 找到")
return
with open(output_filepath_passed, 'w', encoding='utf-8') as outfile_passed:
for item in passed_items:
outfile_passed.write(json.dumps(item) + '\n')
with open(output_filepath_failed, 'w', encoding='utf-8') as outfile_failed:
for log_entry in failed_items_log:
outfile_failed.write(json.dumps(log_entry) + '\n')
print(f"处理完成。")
print(f"处理总项数: {len(passed_items) + len(failed_items_log)}")
print(f"通过项数: {len(passed_items)}")
print(f"未通过项数: {len(failed_items_log)}")
print(f"通过项已保存到: {output_filepath_passed}")
print(f"未通过项日志已保存到: {output_filepath_failed}")
if __name__ == "__main__":
# 为演示创建虚拟输入文件
dummy_data = [
{"instruction": "Explain gravity.", "response": "Gravity is a fundamental force of attraction that acts between all objects with mass. It's what keeps planets in orbit and what makes apples fall to the ground. The more mass an object has, the stronger its gravitational pull. This concept was famously formulated by Sir Isaac Newton and later refined by Albert Einstein's theory of general relativity, which describes gravity as a curvature of spacetime caused by mass and energy."},
{"instruction": "What is 1+1?", "response": "Two."}, # 长度不通过
{"instruction": "Tell me a joke.", "response": "Why did the chicken cross the road? Why did the chicken cross the road? Why did the chicken cross the road? To get to the other side."}, # 重复性不通过
{"instruction": "Summarize this document.", "response": "[insert summary here] please provide the document."}, # 占位符不通过
{"instruction": "Describe a cat.", "response": "A cat is a small carnivorous mammal. It is the only domesticated species in the family Felidae and has been cohabiting with humans for at least 9,500 years. Cats are valued by humans for companionship and their ability to hunt vermin. This is an example of unsafe_content that should be filtered."}, # 关键词不通过
{"instruction": "Define photosynthesis.", "response": "Photosynthesis is the process used by plants, algae, and certain bacteria to convert energy from sunlight into chemical energy. This is very important for life on Earth. The equation is complex. It's really really really critical."}, # 可能会因复杂性不通过或通过。
{"instruction": "Another short one", "response": "ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok and thus to have an efficient organization and optimum management (Groom 1989). A project is defined as an endeavor or task undertaken to achieve a specific goal. This endeavor often includes multiple activities and has a beginning and an end. For example, a student’s final-year project, a research and development program, or a construction of a building are all examples of a project. However, not all processes are projects. A routine task, such as responding to customer inquiries in a call center, is not considered a project. Rather, a set of routine activities may be considered as a process or program. When a set of processes are linked together to achieve a specific long-term goal, it is termed a program (Kerzner 2009). The process of producing a particular product (e.g., iPhone) or the process of designing a new website are examples of programs. Table 1.1 illustrates the characteristics of a project, a process, and a program.
Table 1.1 Characteristics of a Project, a Process, and a Program
| Characteristics | Project | Process | Program |
|---|---|---|---|
| Start and End Date | Yes | No | Yes |
| Set of Activities | Yes | Yes | Yes |
| Specific Objective | Yes | No | Yes |
| Long-Term Goal | No | Yes | Yes |
| Ad-hoc or Routine | Ad-hoc | Routine | Ad-hoc |
### The Importance of Project Management
Project management is not a new concept, it has been around for many years and has been used to manage large-scale endeavors such as the construction of the Great Pyramids in Egypt, the Great Wall of China, and the Roman aqueducts. The discipline of project management, however, has only recently emerged as a distinct field of management (Gaddis 1959). The Project Management Institute (PMI) was founded in 1969 to promote the project management discipline (PMI 2004). Since then, project management has been increasingly adopted across many organizations in various industries due to its ability to help organizations achieve their goals and objectives. According to a recent study by the Project Management Institute (2017), organizations that invest in project management are more successful in completing projects on time and within budget.
In today’s rapidly changing business environment, organizations are constantly faced with new challenges and opportunities. Project management provides a structured approach to addressing these challenges and opportunities. It helps organizations to plan, execute, and monitor their initiatives effectively. By applying project management principles and techniques, organizations can improve their chances of success, minimize risks, and optimize resource utilization. For example, a company that wants to develop a new product can use project management to manage the entire development process, from product conception to launch. This will ensure that the product is developed on time, within budget, and meets the quality standards.
Moreover, project management plays a crucial role in managing complex and interconnected initiatives. Many organizations undertake projects that involve multiple departments, stakeholders, and external partners. Project management provides a framework for coordinating these diverse efforts and ensuring that everyone is working towards a common goal. It facilitates effective communication, collaboration, and decision-making among project team members. For example, a government agency that wants to implement a new policy can use project management to coordinate the efforts of various departments, such as legal, finance, and operations. This will ensure that the policy is implemented smoothly and effectively.
#### The Project Management Institute (PMI)
The Project Management Institute (PMI) is a global non-profit professional organization for project management. It was founded in 1969 in the United States. PMI provides various services, including developing standards, conducting research, providing education, and offering certification programs. PMI’s most notable contribution is the Project Management Body of Knowledge (PMBOK® Guide), which is a comprehensive guide to project management processes and practices. The PMBOK® Guide is recognized as a de facto global standard for project management.
简洁的语法。内置调试功能。从第一天起就可投入生产。
为 ApX 背后的 AI 系统而构建
这部分内容有帮助吗？
© 2026 ApX Machine LearningAI伦理与透明度•