Qwen2ForSequenceClassification文本分类实战和经验分享
本文主要使用Qwen2ForSequenceClassification实现文本分类任务。文章首发于我的知乎：https://zhuanlan.zhihu.com/p/17468021019
一、实验结果和结论
这几个月，在大模型分类场景做了很多实验，攒了一点小小经验。
1、短文本
1）query情感分类，一般不如BERT
ps：结论和，https://segmentfault.com/a/1190000044485544#item-13，基本一致
2、长文本
1）通话ASR转译长文本，BERT截断512不如LLM
[*]LLM没有截断（如果都阶段512，可能效果差不多）
[*]没有对比，BERT进行文本滑动窗口的版本
2）Base v.s. Instruct
[*]数据量小时，Base微调不如Instruct(Instruct模型有对齐税，但是微调数据量小时，效果还是比Base没见过指令微调样本的好)
3）SFT v.s. LoRA
[*]数据量小时（总样本10K以下，每个标签需要视情况而定），SFT微调不如LoRA（SFT调参成本也更大）
3、分类场景的提升方案
1）生成式微调独有
[*]混合同领域相似数据类型不同业务数据，可以提升若干点
[*]数据分布不能差异太大，特别是文本长度，否则混入这种数据反而会让效果下滑（一个平均长度1.2K，一个平均长度5k）
[*]混入比例（接近2：1，不同场景需自行尝试）
[*]混入顺序（我使用的是随机采样，没验证是否分开先后训练顺序是否有影响）
[*]优化提示词（提示词中，增加各类别标签的精要描述；短文本可尝试few-shot）
2）分类头微调 + 生成式微调
[*]数据量大时（10K以上，平均每个标签样本充足），尝试微调Base，而不是微调Instruct
[*]数据增强：尝试无标注数据上跑的伪标签样本（提示词抽的标签 + 微调后的模型抽的标签）
[*]数据量大时，尝试SFT
[*]LoRA微调时，加入LLM的embedding层（未验证过）
[*]尝试蒸馏更大模型到小模型（痛点：大模型难调参，训练成本更高，部署上线还是得小模型）
[*]尝试LoRA的变体
[*]尝试调参（试过optuna自动搜索，效果也不太好；一般就调lr, epoch, rank）
3）重量级
[*]Base模型，领域数据增量预训练后，再进行指令微调
[*]方案待验证
[*]这边尝试了在Qwen2-7B-Instruct的领域数据指令微调后的模型，微调效果反而比直接微调Qwen2-7B-Instruct效果差些。由于不清楚该模型训练步骤细节，所以原因尚不明确）
[*]若验证成功，其好处是训出来的基座在各个领域任务上的微调都能提点
第一优先级，还是搞数据。其次，才是尝试各种方案的加加减减。
4、注意点
[*]学习率
[*]训练的参数量越大，学习率适当要调小
[*]标签噪声
[*]样本标注错误，需要在错误分析时进行剔除和校正
[*]分类业务规则
[*]复杂场景，需要提前确定好完备的标注规则，避免返工（那些模型可以做，那么模型不能做）
二、文本分类-从BERT到LLM
Qwen2ForSequenceClassification和BERTForSequenceClassification，逻辑上是一致的。都是在模型的输出层，加上一个Linear层，用来完成分类任务。
之前在，BERT上做的所有改动，都可以迁移到LLM上。譬如，BERT-CRF。
和BERTForSequenceClassification一样。
BertForSequenceClassification是一个已经实现好的用来进行文本分类的类，一般用来进行文本分类任务。
通过num_labels传递分类的类别数，从构造函数可以看出这个类大致由3部分组成，1个是Bert，1个是Dropout，1个是用于分类的线性分类器Linear。
class BertForSequenceClassification(BertPreTrainedModel): def __init__(self, config): super(BertForSequenceClassification, self).__init__(config) self.num_labels = config.num_labelspython self.bert = BertModel(config) self.dropout = nn.Dropout(config.hidden_dropout_prob) self.classifier = nn.Linear(config.hidden_size, self.config.num_labels) self.init_weights()Bert用于提取文本特征进行Embedding，Dropout防止过拟合，Linear是一个弱分类器，进行分类，如果需要用更复杂的网络结构进行分类可以参考它进行改写。
forward()函数里面已经定义了损失函数，训练时可以不用自己额外实现，返回值包括4个内容
def forward(...): ... if labels is not None: if self.num_labels == 1: #We are doing regression loss_fct = MSELoss() loss = loss_fct(logits.view(-1), labels.view(-1)) else: loss_fct = CrossEntropyLoss() loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) outputs = (loss,) + outputs return outputs# (loss), logits, (hidden_states), (attentions)接下来。看看Qwen2ForSequenceClassification。
Qwen2ForSequenceClassification((model): Qwen2Model( (embed_tokens): Embedding(151936, 1024, padding_idx=151643) (layers): ModuleList( (0-23): 24 x Qwen2DecoderLayer( (self_attn): Qwen2SdpaAttention( (q_proj): Linear(in_features=1024, out_features=1024, bias=True) (k_proj): Linear(in_features=1024, out_features=1024, bias=True) (v_proj): Linear(in_features=1024, out_features=1024, bias=True) (o_proj): Linear(in_features=1024, out_features=1024, bias=False) (rotary_emb): Qwen2RotaryEmbedding() ) (mlp): Qwen2MLP( (gate_proj): Linear(in_features=1024, out_features=2816, bias=False) (up_proj): Linear(in_features=1024, out_features=2816, bias=False) (down_proj): Linear(in_features=2816, out_features=1024, bias=False) (act_fn): SiLU() ) (input_layernorm): Qwen2RMSNorm() (post_attention_layernorm): Qwen2RMSNorm() ) ) (norm): Qwen2RMSNorm())(score): Linear(in_features=1024, out_features=3, bias=False))Qwen2官方代码实现，内置三种模式：
[*]single_label_classification 单标签分类
[*]损失为CrossEntropyLoss
[*]取单标签对应logit，算负对数似然
[*]multi_label_classification 多标签分类
[*]损失为BCEWithLogitsLoss
[*]标签为muilti-hot, 预测logits计算sigmoid，实际取对应维度标签Logit，损失求和
[*]regression 回归
[*]损失为MSELoss
[*]默认为单维度回归（回归可以作为奖励模型，预测打分）
这3种模式的输入标签不同。
class Qwen2ForSequenceClassification(Qwen2PreTrainedModel): def __init__(self, config): super().__init__(config) self.num_labels = config.num_labels self.model = Qwen2Model(config) self.score = nn.Linear(config.hidden_size, self.num_labels, bias=False) # Initialize weights and apply final processing self.post_init() def get_input_embeddings(self): return self.model.embed_tokens def set_input_embeddings(self, value): self.model.embed_tokens = value @add_start_docstrings_to_model_forward(QWEN2_INPUTS_DOCSTRING) def forward( self, input_ids: torch.LongTensor = None, attention_mask: Optional = None, position_ids: Optional = None, past_key_values: Optional] = None, inputs_embeds: Optional = None, labels: Optional = None, use_cache: Optional = None, output_attentions: Optional = None, output_hidden_states: Optional = None, return_dict: Optional = None, ) -> Union: r""" labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in ``. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). """ return_dict = return_dict if return_dict is not None else self.config.use_return_dict transformer_outputs = self.model( input_ids, attention_mask=attention_mask, position_ids=position_ids, past_key_values=past_key_values, inputs_embeds=inputs_embeds, use_cache=use_cache, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) hidden_states = transformer_outputs logits = self.score(hidden_states) if input_ids is not None: batch_size = input_ids.shape else: batch_size = inputs_embeds.shape if self.config.pad_token_id is None and batch_size != 1: raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.") if self.config.pad_token_id is None: sequence_lengths = -1 else: if input_ids is not None: # if no pad token found, use modulo instead of reverse indexing for ONNX compatibility sequence_lengths = torch.eq(input_ids, self.config.pad_token_id).int().argmax(-1) - 1 sequence_lengths = sequence_lengths % input_ids.shape[-1] sequence_lengths = sequence_lengths.to(logits.device) else: sequence_lengths = -1 pooled_logits = logits loss = None if labels is not None: labels = labels.to(logits.device) if self.config.problem_type is None: if self.num_labels == 1: self.config.problem_type = "regression" elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int): self.config.problem_type = "single_label_classification" else: self.config.problem_type = "multi_label_classification" if self.config.problem_type == "regression": loss_fct = MSELoss() if self.num_labels == 1: loss = loss_fct(pooled_logits.squeeze(), labels.squeeze()) else: loss = loss_fct(pooled_logits, labels) elif self.config.problem_type == "single_label_classification": loss_fct = CrossEntropyLoss() loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) elif self.config.problem_type == "multi_label_classification": loss_fct = BCEWithLogitsLoss() loss = loss_fct(pooled_logits, labels) if not return_dict: output = (pooled_logits,) + transformer_outputs return ((loss,) + output) if loss is not None else output return SequenceClassifierOutputWithPast( loss=loss, logits=pooled_logits, past_key_values=transformer_outputs.past_key_values, hidden_states=transformer_outputs.hidden_states, attentions=transformer_outputs.attentions, )三、LoRA微调 Qwen2ForSequenceClassification
在LoRA微调后，将合并LoRA权重，并存储模型。
因为，目前PEFT代码没有，把分类头Linear层的参数存储下来。只靠LoRA权重无法，复现训练的Qwen2ForSequenceClassification模型。
有需要可以小改下代码
这边在modelscope提供的环境，完成了代码的测试。
from modelscope import AutoModelForCausalLM, AutoTokenizermodel_name_or_path = "qwen/Qwen2.5-3B-Instruct"model = AutoModelForCausalLM.from_pretrained(model_name_or_path,torch_dtype="auto",device_map="auto")tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)prompt = "不想学习怎么办？有兴趣，但是拖延症犯了"messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}]text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True)model_inputs = tokenizer(, return_tensors="pt").to(model.device)generated_ids = model.generate( **model_inputs, max_new_tokens=512)generated_ids = [ output_ids for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)]response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)```shellDownloading : 100%|██████████| 661/661 Downloading : 100%|██████████| 2.00/2.00 Downloading : 100%|██████████| 242/242 Downloading : 100%|██████████| 7.21k/7.21k Downloading : 100%|██████████| 1.59M/1.59M Downloading : 100%|██████████| 3.70G/3.70G Downloading : 100%|██████████| 2.05G/2.05G Downloading : 100%|██████████| 34.7k/34.7k Downloading : 100%|██████████| 4.79k/4.79k Downloading : 100%|██████████| 6.71M/6.71M Downloading : 100%|██████████| 7.13k/7.13k Downloading : 100%|██████████| 2.65M/2.65M /usr/local/lib/python3.10/site-packages/accelerate/utils/modeling.py:1405: UserWarning: Current model requires 234882816 bytes of buffer for offloaded layers, which seems does not fit any GPU's remaining memory. If you are experiencing a OOM later, please consider using offload_buffers=True.warnings.warn(面对兴趣与拖延之间的矛盾，确实会让人感到困扰。这里有一些建议或许能帮助你克服拖延，更好地坚持学习：
[*]设定小目标：将大目标分解为一系列小目标。完成每一个小目标都是一次小小的胜利，这可以增加你的动力和成就感。
[*]制定计划：为自己规划一个详细的学习计划，并尽量按照计划执行。记得为休息时间留出空间，保持良好的工作与休息平衡。
[*]保持积极心态：对自己保持耐心和理解，不要因为一时的困难而放弃。记住，进步的过程就是成长的过程。
由于modelscope不支持LORA, 这边查看了本地路径
print(model.model_dir)/mnt/workspace/.cache/modelscope/hub/qwen/Qwen2___5-3B-Instruct查看文件
config.json merges.txt README.mdconfiguration.json model-00001-of-00002.safetensorstokenizer_config.jsongeneration_config.json model-00002-of-00002.safetensorstokenizer.jsonLICENSE model.safetensors.index.json vocab.jsonhuggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...微调代码
### 初始化设定和随机种子import osos.environ["CUDAVISIBLE_DEVICES"] = "0"import torchimport numpy as npimport pandas as pdimport randomseed = 42random.seed(seed)np.random.seed(seed)torch.manual_seed(seed)torch.cuda.manual_seed(seed)torch.backends.cudnn.deterministic = Truetorch.backends.cudnn.benchmark = False基于prompt用大模型构造了20个样本。
import jsonx = '''这基金表现也太差了吧，买了半年了还亏着呢。管理费收得比别的基金都高，感觉就是在给基金公司打工。想查查具体投了啥，结果发现透明度低得要命，啥也看不清楚。基金经理换来换去的，都不知道到底谁在管我的钱。客服电话打过去半天才有人接，问个问题还得等上好几天才有回复。市场稍微有点风吹草动，这基金就跌得比谁都快。投资组合里全是同一行业的股票，风险大得让人睡不着觉。长期持有也没见赚多少钱，还不如存银行定期。分红政策一会儿一个样，根本没法做财务规划。当初宣传时说得好听，实际操作起来完全不是那么回事。'''x_samples = x.split("\n")y = '''这基金真的稳啊，买了之后收益一直挺不错的，感觉很靠谱！管理团队超级专业，每次市场波动都能及时调整策略，让人放心。透明度很高，随时都能查到投资组合的情况，心里有数。基金经理经验老道，看准了几个大机会，赚了不少。客服态度特别好，有问题总能很快得到解答，服务真是没得说。即使在市场不好的时候，这基金的表现也比大多数同类产品强。分散投资做得很好，风险控制得很到位，睡个安稳觉没问题。长期持有的话，回报率真的非常可观，值得信赖。分红政策明确而且稳定，每年都能按时收到分红，计划财务很方便。宣传时承诺的那些好处都实现了，真心觉得选对了这只基金。'''y_samples = y.split("\n")# 创建一个Python字典x_data = [{"content": i, "label": 0, "标注类别": "正向"} for i in x_samples]y_data = [{"content": i, "label": 1, "标注类别": "负向"} for i in y_samples]def save_json(path, data): # 将Python字典转换为JSON字符串 with open(path, 'w', encoding='utf-8') as f: json.dump(data, f, ensure_ascii=False, indent=4)save_json('data/classify_train.json', x_data[:6]+y_data[:6])save_json('data/classify_valid.json', x_data+y_data)save_json('data/classify_test.json', x_data+y_data)数据加载
import jsonfrom tqdm import tqdmfrom loguru import loggerfrom datasets import Dataset, load_datasetdef get_dataset_from_json(json_path, cols): with open(json_path, "r") as file: data = json.load(file) df = pd.DataFrame(data) dataset = Dataset.from_pandas(df, split='train') return dataset# load_dataset加载json的dataset太慢了cols = ['content', 'label', '标注类别']train_ds = get_dataset_from_json('data/classify_train.json', cols)logger.info(f"TrainData num: {len(train_ds)}")valid_ds = get_dataset_from_json('data/classify_valid.json', cols)logger.info(f"ValidData num: {len(valid_ds)}")test_ds = get_dataset_from_json('data/classify_test.json', cols)logger.info(f"TestData num: {len(test_ds)}")
print(train_ds){'content': '这基金表现也太差了吧，买了半年了还亏着呢。', 'label': 0, '标注类别': '正向'}准备dataset(简单实现截断和padding, 无动态padding)
id2label = {0: "正向", 1: "负向"}label2id = {v:k for k,v in id2label.items()}from transformers import AutoTokenizer, DataCollatorWithPadding# from modelscope import AutoTokenizer, DataCollatorwithPaddingmodel_name_or_path = "/mnt/workspace/.cache/modelscope/hub/qwen/Qwen2___5-3B-Instruct"model_name = model_name_or_path.split("/")[-1]print(model_name)tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, padding_side='left')tokenizer.add_special_tokens({'pad_token': '<|endoftext|>'})data_collator = DataCollatorWithPadding(tokenizer=tokenizer)MAX_LEN = 24txt_colname = 'content'def preprocess_function(examples): # padding后处理效率不高，需要动态batch padding return tokenizer(examples, max_length=MAX_LEN, padding=True, truncation=True)tokenized_train = train_ds.map(preprocess_function, num_proc=64, batched=True)tokenized_valid = valid_ds.map(preprocess_function, num_proc=64, batched=True)sklearn评测代码
from sklearn.metrics import ( classification_report, confusion_matrix, accuracy_score, f1_score, precision_score, recall_score)def evals(test_ds, model): k_list = for x in test_ds] model.eval() k_result = [] for idx, txt in tqdm(enumerate(k_list)): model_inputs = tokenizer(, max_length=MAX_LEN, truncation=True, return_tensors="pt").to(model.device) logits = model(**model_inputs).logits res = int(torch.argmax(logits, axis=1).cpu()) k_result.append(id2label.get(res)) y_true = np.array(test_ds['label']) y_pred = np.array() return y_true, y_preddef compute_metrics(eval_pred): predictions, label = eval_pred predictions = np.argmax(predictions, axis=1) return {"f1": f1_score(y_true=label, y_pred=predictions, average='weighted')}def compute_valid_metrics(eval_pred): predictions, label = eval_pred y_true, y_pred = label, predictions accuracy = accuracy_score(y_true, y_pred) print(f'Accuracy: {accuracy}') metric_types = ['micro', 'macro', 'weighted'] for metric_type in metric_types: precision = precision_score(y_true, y_pred, average=metric_type) recall = recall_score(y_true, y_pred, average=metric_type) f1 = f1_score(y_true, y_pred, average=metric_type) print(f'{metric_type} Precision: {precision}') print(f'{metric_type} Recall: {recall}') print(f'{metric_type} F1 Score: {f1}')模型加载，使用Trainer进行训练
import torchfrom transformers import AutoModelForSequenceClassificationfrom transformers import Trainer, TrainingArgumentsfrom peft import get_peft_config, PeftModel, PeftConfig, get_peft_model, LoraConfig, TaskTyperank = 64alpha = rank*2training_args = TrainingArguments( output_dir=f"./output/{model_name}/seqence_classify/", learning_rate=5e-5, per_device_train_batch_size=8, per_device_eval_batch_size=4, num_train_epochs=3, weight_decay=0.01, evaluation_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True)peft_config = LoraConfig( task_type=TaskType.SEQ_CLS, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], inference_mode=False, r=rank, lora_alpha=alpha, lora_dropout=0.1)model = AutoModelForSequenceClassification.from_pretrained( model_name_or_path, num_labels=len(id2label), id2label=id2label, label2id=label2id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, attn_implementation="flash attention2")model.config.pad_token_id = tokenizer.pad_token_idmodel = get_peft_model(model, peft_config)model.print_trainable_parameters()trainer = Trainer( model=model, args=training_args, train_dataset=tokenized_train, eval_dataset=tokenized_valid, tokenizer=tokenizer, data_collator=data_collator, compute_metrics=compute_metrics)logger.info(f"start Trainingrank: {rank}")trainer.train()logger.info(f"Valid Set, rank: {rank}")y_true, y_pred = evals(valid_ds, model)metrics = compute_valid_metrics((y_pred, y_true))logger.info(metrics)logger.info(f"Test Set, rank: {rank}")y_true, y_pred = evals(test_ds, model)metrics = compute_valid_metrics((y_pred, y_true))logger.info(metrics)saved_model = model.merge_and_unload()saved_model.save_pretrained('/model/qwen2-3b/seqcls')将LoraConfig和get_peft_model去掉，就是SFT的代码。
model的结构
PeftModelForSequenceClassification((base_model): LoraModel( (model): Qwen2ForSequenceClassification( (model): Qwen2Model( (embed_tokens): Embedding(151936, 2048) (layers): ModuleList( (0-35): 36 x Qwen2DecoderLayer( (self_attn): Qwen2SdpaAttention( (q_proj): Linear(in_features=2048, out_features=2048, bias=True) (k_proj): Linear(in_features=2048, out_features=256, bias=True) (v_proj): Linear(in_features=2048, out_features=256, bias=True) (o_proj): Linear(in_features=2048, out_features=2048, bias=False) (rotary_emb): Qwen2RotaryEmbedding() ) (mlp): Qwen2MLP( (gate_proj): Linear(in_features=2048, out_features=11008, bias=False) (up_proj): Linear(in_features=2048, out_features=11008, bias=False) (down_proj): Linear(in_features=11008, out_features=2048, bias=False) (act_fn): SiLU() ) (input_layernorm): Qwen2RMSNorm((2048,), eps=1e-06) (post_attention_layernorm): Qwen2RMSNorm((2048,), eps=1e-06) ) ) (norm): Qwen2RMSNorm((2048,), eps=1e-06) ) (score): Linear(in_features=2048, out_features=2, bias=False) )))预测
txt = "退钱，什么辣鸡基金"model_inputs = tokenizer(, max_length=MAX_LEN, truncation=True, return_tensors="pt").to(saved_model.device)logits = saved_model(**model_inputs).logitsres = int(torch.argmax(logits, axis=1).cpu())print(id2label)负向output输出类型
SequenceClassifierOutputWithPast(loss=None, logits=tensor([[-0.1387,2.3438]], device='cuda:0', grad_fn=<IndexBackward0>), past_key_values=((tensor([[[[ -3.3750, 0.3164, 2.3125,...,56.5000,26.0000,87.0000], [ -4.6875, 3.0312, 0.6875,...,57.7500,24.3750,86.0000], [ -0.7109, 1.1094,-0.7383,...,56.7500,24.8750,86.5000], ..., ..., [-0.2188,0.2148,0.4375,..., -0.1016,0.9336, -1.1016], [ 1.3281,0.3359,1.3125,..., -0.3906,0.0312, -0.0391], [ 0.8789,0.5312,1.4297,...,0.1797, -0.9609, -0.6445]]]], device='cuda:0'))), hidden_states=None, attentions=None)跑测结果
Some weights of Qwen2ForSequenceClassification were not initialized from the model checkpoint at /mnt/workspace/.cache/modelscope/hub/qwen/Qwen2___5-3B-Instruct and are newly initialized: ['score.weight']You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.Detected kernel version 4.19.91, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.2024-10-09 23:54:07.615 | INFO | __main__:<module>:53 - start Trainingrank: 64trainable params: 119,738,368 || all params: 3,205,681,152 || trainable%: 3.7352 Epoch Training Loss Validation Loss F11 No log 0.988281 0.3333332 No log 0.527344 0.7333333 No log 0.453125 1.0000002024-10-09 23:54:17.371 | INFO | __main__:<module>:56 - Valid Set, rank: 644it 2024-10-09 23:54:17.896 | INFO | __main__:<module>:59 - None2024-10-09 23:54:17.897 | INFO | __main__:<module>:61 - Test Set, rank: 64Accuracy: 1.0micro Precision: 1.0micro Recall: 1.0micro F1 Score: 1.0macro Precision: 1.0macro Recall: 1.0macro F1 Score: 1.0weighted Precision: 1.0weighted Recall: 1.0weighted F1 Score: 1.04it 2024-10-09 23:54:18.218 | INFO | __main__:<module>:64 - NoneAccuracy: 0.75micro Precision: 0.75micro Recall: 0.75micro F1 Score: 0.75macro Precision: 0.8333333333333333macro Recall: 0.75macro F1 Score: 0.7333333333333334weighted Precision: 0.8333333333333333weighted Recall: 0.75weighted F1 Score: 0.7333333333333334四、自测结果
短文本
常用的对话中，客户的单轮query, 情感极性分类。
3分类，长度最大128字，训练样本量6K左右
query ，比不过基础的BERTAccuracy:0.9334389857369255microPrecision:0.9334389857369255microRecall:0.9334389857369255micro F1Score:0.9334389857369255macro Precision:0.9292774942877138macro Reca1l:0.9550788300142491macro F1Score:0.9388312342456646weightedPrecision:0.9418775412386249weighted Recall:0.9334389857369255weighted F1Score:0.93383533375322 precision recall fi-score support0 1.00 0.88 0.93 3341 0.94 0.99 0.97 1012 0.85 0.99 0.92 196accuracy 0.93macro avg 0.93weightedavg 0.94 使用Chinese-RoBerta-large-wwm，及其各种变体进行比较。7B、3B、1.5B和0.5B，均无优势。比不过Large、Base。一些裁减参数量就几十M的都能到85左右，所以看不出LLM的优势。
结论：
短文本场景，LLM的优势在于少样本、样本不均匀，以及基于prompt+fewshot用72B规模生成伪标签。
除非样本量上万，且价值比较大的场景，可以尝试14B以上模型，调参确认提点后，再进行蒸馏。
绝大部分短文本场景没有必要用到大模型，除非是生成场景，比如query扩写、query多轮改写。
长文本
这里用到的是ASR转译文本
训练集4918样本，平均长度740字，最大4631字，75% 918字
LoRA微调结果，一般2个epoch效果好些，rank要适当调参。
epoch=1， rank=96, alpha=2*rank
Accuracy:0.8415637860082305micro Precision:0.8415637860082305 micro Recall:0.8415637860082305micro F1 Score:0.8415637860082305macro Precision:0.8075007129137883macro Recall: 0.770659344467927macroF1 Score:0.7726373117446225weightedPrecision:0.8509932419375813weighted Recall:0.8415637860082305weighted F1Score:0.8420807262647815 precision recall f1-score support0 0.95 0.83 0.89 1631 0.76 0.77 0.77 662 0.78 0.89 0.83 633 0.81 0.81 0.81 424 0.80 0.93 0.86 305 0.48 0.56 0.51 186 1.00 0.43 0.60 77 0.88 0.95 0.92 97epoch=3，rank=96, alpha=2*rank
Accuracy:0.8847736625514403micro Precision:0.8847736625514403micro Recall:0.8847736625514403micro F1 Score:0.8847736625514403 macro Precision:0.8765027065399982macroRecall:0.8400805218716799macro F1 Score:.8527883278910355weighted Precision:0.8903846924862034weighted Recall:0.8847736625514403weighted F1 Score:0.8852820009557909 precision recall fl-score support0 0.94 0.89 0.91 1631 0.77 0.85 0.81 662 0.81 0.88 0.83 423 0.79 0.90 0.86 634 1.00 0.93 0.97 305 0.92 0.61 0.73 186 0.83 0.71 0.77 77 0.96 0.94 0.95 97相关资料
[*]比较详细的LLM分类头微调经验
https://zhuanlan.zhihu.com/p/704983302
[*]在灾难推文分析场景上比较用 LoRA 微调 Roberta、Llama 2 和 Mistral 的过程及表现
https://segmentfault.com/a/1190000044485544
[*]SFT分类头微调代码（其实就是去掉LoRA那几行代码）
https://github.com/muyaostudio/qwen2_seq_cls
[*]知乎上的一个代码
https://zhuanlan.zhihu.com/p/691459595
相关代码可以在github上找找，kaggle也推荐去，主要就是这2个地方。
页:
[1]