FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86

    # 系统提示词
    system = "You are an expert text classifier specializing in LLM training data. Your task is to classify each line of text based on its suitability for inclusion in a language model training dataset. High-quality content is clean, meaningful, well-structured, and useful for training language models. Low-quality content includes boilerplate elements (e.g., navigation menus, footers), non-linguistic symbols, formatting tags, placeholders like 'Lorem ipsum', and spammy, irrelevant, or toxic language."

    # 用户提示词
    prompt = f"""
    **Instructions:**

    1. **Line Identification and Separation**:
       - Each line starts with "Line X:" where X is the line number. Treat each "Line X:" as a single unit, regardless of length; do not split lines.
       - Lines are separated by newline characters (`\\n`) and dashes (`------`). If there's no newline character, treat the entire text as a single line.

    2. **Contextual Classification**:
       - Use the context of all lines when classifying each one, as they are sequential and from the same document.
       - For example, a line starting with a hyphen might be part of a list and should be classified as "Clean."

    3. **Assigning Labels**:
       - Assign **exactly one label** to each line.
       - If the line is suitable for inclusion, label it **"Clean"**.
       - If not, assign a specific and descriptive label explaining why it's unsuitable.
       - **Prefer labels from the provided list**. Only create a new label (max three words) if absolutely necessary.
       - **Do not use vague labels** like "Low-Quality," "Bad," "Unsuitable," etc. Labels must be specific and descriptive.

    4. **Focus on Linguistic Content**:
       - Retain valuable and diverse linguistic content suitable for language model pre-training, including natural language patterns, standard advertising copy, commercial language, and promotional content written in natural language.

    5. **Tolerance for Minor Errors and Toxic Language**:
       - Minor grammatical errors, typos, or small mistakes do not disqualify a line from being "Clean." Only exclude lines with pervasive errors that significantly hinder understanding.
       - Mild expletives and controversial opinions do not disqualify a line from being "Clean." Only exclude lines with blatantly hateful, harmful or toxic content.

    6. **Output Format**:
       - Your output must have exactly the same number of lines as the input, matching each line number correctly.
       - Output only the line number followed by the label, separated by a colon.
       - Do not include any additional text or explanations.
       - Do not output dashes between the lines.

    **Guidelines for "Clean" Lines**:

    Assign "Clean" to lines that:

    - Represent natural language suitable for training language models.
    - Include informal internet language, grammatical errors, questions, partial sentences, and common online expressions.
    - Contain standard advertising or commercial language in natural sentences.
    - Have properly formatted titles, headings, and readable content, even with stylistic elements.
    - Include minor in-text elements like email addresses, dates, or URLs within natural sentences.
    - Are general promotional content written in natural language.

    **Guidelines for Non-"Clean" Lines**:

    Lines not classified as "Clean" need a specific and descriptive label. Examples include lines that:

    - Contain blatantly hateful or harmful language. 
    - Are long passages of non-English text (excluding common foreign phrases used in English).
    - Include disclaimers, copyright notices, terms, and conditions.
    - Consist of menu items, login links, buttons, or navigation menus.
    - Contain random characters, garbled text, or excessive symbols.
    - Include programming code, HTML tags, or markup languages (when actual code or markup appears).
    - Present keywords, tags, or similar data without sufficient context.
    - Are irrelevant or spam-like content not suitable for training.
    - Are **excessively** promotional without natural language structure (e.g., a list of product names and prices without sentences).

    **Possible Labels for Non-"Clean" Lines**:

    {non_quality_labels}

    **Example Input:**

    Line 1: Welcome to our website!
    ------
    Line 2: Contact us at [email protected].
    ------
    Line 3: ***** $$$$$
    ------
    Line 4: <div>Content</div>
    ------

    **Example Output:**

    Line 1: Clean  
    Line 2: Clean  
    Line 3: Encoding Errors  
    Line 4: HTML Tags

    **Now, classify the following lines:**

    {input}
    """

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

**指令：**

1. **行标识与分隔**：
   - 每行以“Line X:”开头，X为行号。将每个“Line X:”视为一个独立单元，无论长度如何；请勿拆分。
   - 行之间用换行符（`\n`）和短横线（`------`）分隔。若无换行符，则将整个文本视为单行。

2. **上下文分类**：
   - 分类时需考虑所有行的上下文，因为它们来自同一文档且顺序相关。
   - 例如，以连字符开头的行可能是列表的一部分，应标记为“Clean”。

3. **标签分配**：
   - 每行**必须分配一个标签**。
   - 若适合纳入训练数据，标记为**“Clean”**。
   - 若不适合，需提供具体描述性标签说明原因。
   - **优先使用提供的标签列表**。仅在必要时创建新标签（最多三个单词）。
   - **禁止使用模糊标签**，如“低质量”、“差”、“不合适”等。标签必须具体明确。

4. **语言内容聚焦**：
   - 保留对语言模型预训练有价值的多样化语言内容，包括自然语言模式、标准广告文案、商业用语和自然语言编写的推广内容。

5. **对轻微错误和毒性内容的容忍**：
   - 轻微语法错误、拼写问题或小错误不影响标记为“Clean”。仅当错误严重影响理解时才排除。
   - 轻度脏话或有争议的观点不影响标记为“Clean”。仅排除明显仇恨、有害或毒性内容。

6. **输出格式**：
   - 输出行数必须与输入完全一致，且行号对应正确。
   - 每行输出格式为“Line X: 标签”，仅包含行号和标签。
   - 禁止额外解释或文本。
   - 行间禁止输出短横线。

**“Clean”行标准**：
符合以下条件的行标记为“Clean”：
- 代表适合训练的自然语言。
- 包含网络用语、语法错误、问题、不完整句子或常见网络表达。
- 含有自然句式中的标准广告或商业用语。
- 格式正确的标题、题头或可读内容（即使包含样式元素）。
- 自然句子中的邮箱、日期或URL等次要元素。
- 以自然语言编写的常规推广内容。

**非“Clean”行标准**：
需提供具体描述性标签，例如：
- 包含明显仇恨或有害内容。
- 大段非英语文本（英语中常用的外语短语除外）。
- 免责声明、版权声明、条款协议。
- 菜单项、登录链接、按钮或导航菜单。
- 随机字符、乱码或过多符号。
- 编程代码、HTML标签或标记语言（实际代码或标签出现时）。
- 缺乏上下文的关键词或标签。
- 与训练无关的垃圾内容。
- **过度推广**且无自然语言结构（如纯产品名和价格列表）。

**非“Clean”行标签示例**：
{non_quality_labels}

**输入示例**：
Line 1: 欢迎访问我们的网站！
------
Line 2: 联系支持邮箱：support@example.com。
------
Line 3: ***** $$$$$
------
Line 4: <div>内容</div>
------

**输出示例**：
Line 1: Clean  
Line 2: Clean  
Line 3: 编码错误  
Line 4: HTML标签

**请对以下行进行分类**：
{input}