Group Preference Aligner

Advancements in Large Language Models (LLMs) have highlighted the critical need for aligning their outputs with the diverse preferences of user groups. The risks of bias and misrepresentation should be addressed. Traditional methods such as supervised fine-tuning and strategic prompting often fall short in terms of efficiency or effectiveness. Our research introduces a novel Group Preference Aligner (GPA) model that significantly enhances group-specific alignment by leveraging a two-stage process. Initially, we use a universal aligner to adjust LLM outputs towards universally beneficial attributes like safety and harmlessness. Subsequently, we fine-tune this model using only adapter modules tailored to the specific preferences of individual groups, enhancing the LLM's ability to cater to nuanced group characteristics, while increasing the parameters marginally. We introduce a new synthetic DescriptiveOpinionQA dataset to model more descriptive group preferences beyond multiple-choice options in the OpinionQA dataset. We demonstrate the effectiveness of our approach using the OpinionQA and our synthetic DescriptiveOpinionQA datasets. Our findings show that the GPA model aligns more closely with group-specific preferences compared to existing models.

LLMs frequently exhibit biased representations, disproportionately emphasizing or neglecting certain groups. Directive prompts like "Speak as [specific identity or role]" offer a partial solution by providing contextual cues, yet they risk misrepresentation and have inherent limitations. Historically, methods such as supervised fine-tuning, strategic prompting, and learning from contextual examples have been used to align LLM outputs with user or group preferences. These methods, however, often prove to be inefficient or ineffective.

Group Preference Optimization

The recent Group Preference Optimization (GPO) approach tries to address these challenges. GPO leverages a framework that aligns LLM outputs with the preferences of various groups using a few-shot learning paradigm. This method uses an auxiliary transformer module that predicts the preferences of a group based on the outputs of a base LLM among the options provided.

Group alignment in GPO aims to steer pretrained LLMs to preferences catering to a wide range of groups. For each group g, the preference dataset is represented as D_g = {(x₁^g, y₁^g), ..., (x_n^g, y_n^g)}. Here, y_i^g signifies the preference of group g for a pair of given prompt q_i^g and response r_i^g, while x_i^g is its LLM representation obtained with π_emb(q_i^g, r_i^g), as shown in Figure 1.

As shown in Figure 2 and 3, a set of few known input-output pairs from a group, (x₁, y₁), (x₂, y₂), ..., (x_m, y_m) are provided as the context points along a base prompt. These provide the model with known preferences for the given inputs. Preferences ŷ_m+1, ..., ŷ_n for new inputs (x_m+1, 0), ..., (x_n, 0) are predicted based on the patterns learned from the context points.

Despite its advantages, GPO has limitations. GPO predicts distribution of options for a given question and not the direct answer. This makes it difficult to directly use it as an LLM or as an extension to LLM. GPO is a few-shot paradigm, depending heavily on the context and thus can struggle with longer context lengths.

Aligner

Parallel to GPO, there is another line of work: Aligner. The Aligner model represents a novel approach to aligning LLMs with human opinions without the need for Reinforcement Learning from Human Feedback (RLHF) processes. It doesn't rely on reward model training and actor-critic engineering. It operates on the principle of learning correctional differences between aligned and unaligned responses directly from the data, structured as an autoregressive sequence-to-sequence (seq2seq) model trained on query-answer-correction (Q-A-C) triples. Given a question and output of an LLM as Answer, Aligner predicts the potential correction that is more aligned to safety and harmlessness.

The Aligner model stacks upon an upstream LLM. This model corrects the answers of the upstream LLM's output and redistributes the initial answers, thus aligning the composed LLM responses towards the aligned distribution. Aligner takes the user’s query x and the initial answer y_o generated by the upstream LLM, then generates the answer y_c which is better aligned as required. The seq2seq model is trained to redistribute the preliminary answers y_o to the aligned answer y_c as shown in Figure 4 and 5.

Aligner can be integrated with any pre-existing LLM making it model agnostic. This Plug-and-Play capability enhances its alignment capabilities without the need for direct modifications to the underlying LLM. Aligner demonstrates significant improvements in metrics such as helpfulness and harmlessness, achieving these gains with lower computational demands compared to traditional RLHF methods.

Group Preference Aligner

Building upon the foundational work of the Aligner model, we introduce an innovative extension, Group Preference Aligner (GPA), that significantly enhances group-specific alignment for LLMs. While Aligner focuses on adjusting LLM outputs to generate safer and more harmless responses universally, our Group Aligner is also tailored to the nuances and preferences of individual groups. Specifically, we propose a two-stage alignment process: The first stage involves training a universal aligner as mentioned in the aligner paper to address common preferences across groups (such as safety and harmlessness) and the second stage involves fine-tuning the model by incorporating adapter modules to align with the nuanced preferences of individual groups.

Comparative Advantages of Group Preference Aligner:

• Robustness and Context Length: Unlike GPO, our method support descriptive conversations, and doesn't require longer contexts.
• Efficiency in Group-Specific Preferences: While the first stage has high parameters, our method efficiently manages individual group adaptations with significantly fewer parameters than full fine-tuning approaches, making it highly effective and scalable.
• Plug-and-Play Flexibility: Retaining Aligner’s plug-and-play capability, once trained, our Group Aligner module can be potentially added to any LLM to refine outputs, effectively enhancing model versatility and applicability across various applications.

Datasets

We use the below two datasets.

• Survey Dataset - OpinionQA: OpinionQA is a recent survey dataset designed to facilitate research in opinion-based question answering systems. It comprises a diverse collection of survey questions sourced from real-world data, capturing a wide range of opinions across various demographics. OpinionQA spans 22 US demographic groups (e.g., income, political ideology, race, and sex) across 500 multiple-choice questions.
• Synthetic Dataset - DescriptiveOpinionQA: Since OpinionQA is constructed based on the survey questions which have options, we looked out for aligning conversations that have better descriptive answers rather than simple options. To achieve this, we construct a synthetic dataset using GPT-3.5-Turbo to generate a descriptive version for the same questions available in the OpinionQA dataset. We use prompts similar to those used in steering the LLM in OpinionQA to generate descriptive responses as alternatives to the options provided in the survey question of OpinionQA.

The datasets are processed to get to question-answer-correction (Q-A-C) format following Aligner. Q is the question that is directly picked from OpinionQA. A is the overall distribution answer. C is the corrected response that the Aligner is supposed to generate and align the answer A to the respective group.

OpinionQA

An example question answer distribution from the dataset is shown below.

Question: How much, if at all, do you think the ease with which people can legally obtain guns contributes to gun violence in the country today?

1. A great deal
2. A fair amount
3. Not too much
4. Not at all

Response Distribution

Since the OpinionQA dataset provides distribution statistics of the options selected at the group level, and the overall statistics; we select the answer with high probability from the overall group as the answer from the LLM. The correction for each group is the option with high probability from the answer distribution for the respective question. This would let Aligner correct the responses to generate group opinions.

An example question-answer-correction generated for Democrat and Republican groups is shown below.

Question: How much, if at all, do you think the ease with which people can legally obtain guns contributes to gun violence in the country today?
Overall Answer: A great deal
Correction (Democrat): A great deal
Correction (Republican): Not too much
Correction (Independent): A fair amount

DescriptiveOpinionQA

To generate better descriptive responses, we leveraged GPT-3.5-Turbo and constructed synthetic answers and corrections for each group. We use the following group level prompts and get a response for each question available in the OpinionQA dataset.

group_intepretation_map = {
  "Overall": "Consider yourself as a United States citizen.",
  "Democrat": "Consider yourself as a United States citizen. In today's United States politics, you are Democrat.",
  "Republican": "Consider yourself as a United States citizen. In today's United States politics, you are Republican.",
  "$100,000 or more": "Consider yourself as a United States citizen. Your total family annual income from all sources before taxes is $100,000 or more.",
  "Less than $30,000": "Consider yourself as a United States citizen. Your total family annual income from all sources before taxes is less than $30,000."
}

Data generation follows this sequence:

1. Set the appropriate group context based on the group interpretation map.
2. Generate responses using GPT-3.5-Turbo by feeding a question and interpreting the context.
3. Store the generated responses for each group to all the questions.

system_prompt = f"""
    {group_intepretation_map[Group]} 
    Answer this survey question from your perspective in a brief sentence.
    Don't start the answer with your description or group affiliation.
    The answer should reflect your concerns and life experiences relevant to the topic."
"""
user_prompt = f"Question: {Question}."

chat_completion = openai_client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_prompt}
    ],
    temperature=0.5
)
response = chat_completion.choices[0].message.content

The responses generated for the question in the Overall group is considered as the default response (answer) from the language model, followed by the responses for individual groups as the corrections to the overall answers.

An example question-answer-correction generated for Democrat and Republican groups is shown below.

Question: How much, if at all, do you think the ease with which people can legally obtain guns contributes to gun violence in the country today?
Overall Answer: I believe that the ease with which people can legally obtain guns in the United States does contribute to gun violence, and I am concerned about the impact it has on the safety of our communities.
Correction (Democrat): The ease of legally obtaining guns in the United States definitely contributes to the high rates of gun violence we see in our country today, and stricter gun control measures are necessary to address this issue.
Correction (Republican): I believe that the ease of legally obtaining guns should be balanced with ensuring proper background checks and mental health evaluations to prevent gun violence in our country.

Discussion on Chosen Groups

For our project, we chose specific groups such as income (Less than $30,000, $100,000 or more) and political orientation(Democrat, Republican) based on their disparate nature, as demonstrated in the OpinionQA data analysis.

Group Preference Aligner

Our method builds on the Aligner model. It has two stages.

1. Universal Alignment: Initially, a universal aligner model is trained to address common preferences across groups, focusing on universally applicable aspects such as safety and harmlessness. For this project, we used a pretrained Aligner-7B model. This model was pretrained on 20K aligner (Q-A-C) dataset that was constructed using prompted models like GPT-4, LLama2-70B-Chat and also involving human annotators ensuring the corrections align with established principles emphasizing safety, helpfulness, and harmlessness.
2. Group-Specific Fine-Tuning: The above model is fine-tuned by integrating adapter modules tailored to the specific preferences of individual groups, allowing for precise alignment with nuanced group characteristics. We work with two different settings.
   • In the first case, we take the Aligner model with a single LoRA adapter module and finetune it with data from all the groups.
   • In the second case, we finetune separate LoRA adapter modules for each group.
   We train the adapters on the generated DescriptiveOpinionQA dataset, which has (Q, A, C) datapoints for each group as expected by Aligner. We update the default helpful and harmless correction prompt used in Aligner to the group-specific prompt by instructing the model to generate responses from the viewpoint of an individual from the group. This is changing the correction prompt from "Edit the following Question-Answer pair to make it more helpful and harmless:" to "Edit the following Question-Answer pair to make it more aligned to a Democrat's view:" for the Democrat group.

At the end, we will have one base model and a single adapter module or one base model and multiple adapter modules for each group.

Comparison to GPO

Since GPO requires options to be passed along with the question as an input, and it also predicts the option distribution rather than generating the answer directly, this is not directly comparable to our GPA methodology. Along with this, GPO doesn't have any base model that is trained on safety data. For a better and fair comparison of GPO with our method, we adopted the following training strategies for GPA and GPO.

1. GPA-Objective: We use the same training strategy for GPA as mentioned earlier, but now train it to predict the options available in OpinionQA instead of generating completely descriptive responses. This means we train another GPA model directly on the OpinionQA dataset instead of our generated DescriptiveOpinionQA dataset. For the (Q, A, C) training pairs, the survey question is given as the Q, the answer from the overall distribution with maximum distribution is the A, and the group-specific answer with maximum probability as the correction C. This model would still take only the question as input and not all the options, unlike GPO, but the generated output will be trained to match the option answer instead of a descriptive answer.
2. S-GPO (Safe GPO): Our GPA has a universal alignment stage first that aligns the LLM model towards safety and harmlessness. Following the same methodology, choosing the base LLM that is trained on safety and harmlessness as the first stage before GPO would make the comparison even. The Aligner model was compared to a finetuned alpaca model following the DPO and PPO methodologies trained on the same Aligner dataset. So, we use this DPO finetuned alpaca model as the universal alignment stage before GPO. However, the checkpoints for these are not made available. We replicated the same by finetuning alpaca-7b model on aligner-20K safety dataset using DPO, which serves as the base for the second stage. For the second stage, we now train GPO on this DPO finetuned alpaca model using OpinionQA dataset following the same training strategy used in GPO. For final comparison to GPA-Objective, we convert the distribution of options that's generated by the S-GPO model to a single answer by taking the option that has the highest probability.

By following the above changes to training strategies to start from a similar first stage base, the second stage training should give a good idea about the differences among the GPA and GPO approaches.

Experimental Setup

We use NVIDIA Quadro RTX 8000 and NVIDIA A100 GPUs available on NYU Greene HPC. To finetune Aligner models on OpinionQA and DescriptiveOpinionQA we use the aligner codebase. We use the same codebase to finetune alpaca-7b on the aligner-20K dataset using DPO. To train GPO we follow the code provided at .

Specifically,

• We use aligner codebase for
• Second stage of both GPA-Objective and GPA-Desriptive models.
• First stage of S-GPO: Finetuning alpaca-7b on the aligner-20K dataset using DPO.
• We use GPO codebase for Second stage of S-GPO: Finetuning s-alpaca on Opinion QA using GPO (S-GPO).

Evaluation Metrics

We evaluate the performance of all the trained models through qualitative and quantitative estimates. For qualitative evaluation, we manually evaluate generated responses on the evaluation set and compare its alignment with the correction compared to the answer. For quantitative evaluation, we chose the Rouge metric due to its relevance in assessing the quality of descriptive outputs. The Rouge metric measures the overlap of n-grams between the generated responses and a set of reference responses, providing a robust indicator of textual similarity. It is important to note, however, that while the ROUGE metric offers a useful estimate of similarity, it is not the ideal metric for evaluating all aspects of response quality. Particularly for more descriptive answers, this value may be lower, indicating that ROUGE can sometimes fail to capture the nuanced content and creativity of the responses.

GPA-Descriptive

Following are some qualitative examples from the individual adapters:

Question: Thinking about long-range foreign policy goals, how much priority, if any, do you think reducing legal immigrations into the US should be given?

	Democrat	Republican	Overall Answer
Groundtruth Correction	Reducing legal immigration should not be a top priority in long-range foreign policy goals; instead, the focus should be on comprehensive immigration reform that promotes fairness, security, and economic growth.	Reducing legal immigration should be a significant priority in our long-range foreign policy goals to ensure the protection of American jobs and national security.	Reducing legal immigration should not be a top priority in long-range foreign policy goals. It is important to balance national security concerns with the contributions and diversity that immigrants bring to the United States.
Generated Correction	Reducing legal immigration should not be a top priority in long-range foreign policy goals; instead, I believe in inclusive and fair immigration policies that benefit the United States and all its citizens.	Reducing legal immigration should be a top priority in order to protect American jobs and culture.

Question: How much, if at all, do you think the following proposals would do to reduce economic inequality in the U.S.? Expanding government benefits for the poor.

	Less than $30,000	$100,000 or more	Overall Answer
Groundtruth Correction	Expanding government benefits for the poor would greatly help reduce economic inequality in the U.S. as it would provide much-needed support for low-income families like mine to meet basic needs and have a better chance at financial stability.	Expanding government benefits for the poor could help reduce economic inequality by providing necessary support for those in need, but it is important to ensure that these programs are efficient and effective in targeting those who truly need assistance.	Expanding government benefits for the poor would help reduce economic inequality by providing necessary support for those in need and helping to level the playing field for all citizens.
Generated Correction	Expanding government benefits for the poor would greatly help reduce economic inequality by providing much-needed support to families like mine who are struggling to make ends meet due to limited income.	Expanding government benefits for the poor would definitely help reduce economic inequality by providing much-needed support and opportunities for upward mobility for those struggling to make ends meet.

From the above examples we can see that the generated corrections for each of the groups follow the expected tone as in their groundtruth response.

Rouge scores for GPA trained on DescriptiveOpinionQA dataset for individual adapters trained separately for each group and a common adapter trained collectively for all groups can be seen in the figure below. We follow a train-test split of 90%-10%.

The results indicate comparable performance between the single adapter setting and individual adapters, suggesting that adding more parameters could further enhance the results.

Comparative Analysis

We compare the performance of GPA-Objective with GPO methods and the quantitative results for these as shown below. We see the rouge score of GPA-Objective is better than S-GPO.

Key Highlights

Training and Evaluation Dynamics of GPO: When training GPO by splitting groups into train and eval (90% of groups training, 10% of groups evaluation), the rouge scores and alignment scores (of the predicted distributions, calculated using Wasserstein Distance) are notably high in comparison to taking eval split from all the groups, for both with 22 groups and with a subset of groups. This suggests potential overfitting to familiar questions. The below plots showing the rouge score and alignment score clearly highlights this.

Classification vs. Regression Discrepancies: The evaluation may inherently favor the GPO model as it predicts a probability distribution over options rather than generating direct text. This setup tends to yield higher Rouge scores for GPO, where direct text generation introduces more variability and challenge in matching the exact content of options.

Adaptation to New Groups: Scalability poses a challenge across various alignment models, including our own. Adding a new group to our model necessitates retraining the adapter module, introducing scalability concerns. However, this overhead is manageable, especially when considering the overall benefits of precise group alignment and the fact that the added parameters are minimal compared to the actual size of the Aligner model. While GPO significantly reduces some aspects of scalability issues, it still incurs a small overhead in managing long prompts. It requires the pre-computation of embeddings for all Q&As at the outset. During inference, this necessitates generating context by appending embeddings for a selected number of Q-As each time a new group is assessed. Despite these challenges, both models manage the added complexity with relatively low overhead, maintaining usability and effectiveness across various settings.

Data Generation Challenges

Inconsistent Output Formats: The outputs from LLMs did not adhere to a common format, necessitating extensive post-processing to shape the data into a usable form for training. This involved significant cleaning and manipulation to meet the desired data format.

Temperature Tuning: Adjusting the generation temperature of the LLM was crucial to balance creativity and relevance in the responses, ensuring that the data generated met our expectations.

Prompt Engineering: Crafting the right prompts was a big challenge. We needed to avoid generic responses that could apply uniformly across all groups as well as overly scripted or obvious answers, such as "Given I am a democrat, according to my view, ...". Finding the precise phrasing to elicit useful and varied responses required extensive experimentation.

Sparse Data for Each Group: The limited amount of training data available for each group made it impractical to train models like Aligner from scratch. Thus, we were focused primarily on fine-tuning existing models.

Handling Incomplete Sentences: We encountered data examples with incomplete sentences. These cause trouble in training aligner given it doesn't know the full question to answer as expected by the ground truth. Examples include:

{
  "key": "LEGALIMG_W41",
  "question": "In order to maintain the strength of the U.S. economy over the next 30 years, do you think that legal immigration will need to be",
  "answer": "Maintained at current levels",
  "correction": "Maintained at current levels"
},
{
  "key": "LOCALELECT_W29",
  "question": "The next question is about local elections, such as for mayor or a school board. Do you",
  "answer": "Always vote in local elections",
  "correction": "Never vote in local elections"
}

Hardware and Computational Challenges

Out-of-Memory Issues:

• The size of the models often exceeded the available GPU memory, posing significant challenges in managing model loads and operations.
• Managing the maximum generation length without exceeding memory capacities required experimentation too.

Exploring DeepSpeed Zero Stages: We had to experiment with different DeepSpeed Zero optimization stages to find the right balance between memory usage and computational time.

Precision and Efficiency: To manage these large models effectively, we loaded them in lower bit resolutions (e.g., using 16-bit precision) for more memory-efficient training, while maintaining calculations in 32-bit to preserve accuracy.

• Enhanced Evaluation Metrics: Incorporate language models to supplement the ROUGE metrics for a more comprehensive evaluation of model performance.
• Data Diversity and Scale: Develop methodologies to automatically generate additional questions based on existing datasets, thereby enriching the dataset to cover a broader range of group preferences. Investigate the impact of utilizing larger and more varied datasets to enhance the robustness and generalizability of our findings.
• Other Alignment Strategies: Experiment with techniques to perform a direct one-stage group preference alignment and compare its efficiency against the two-stage approach proposed in our study. Limited dataset size has previously restricted this analysis, but with access to more comprehensive data, a detailed comparison could be achievable. We can also experiment with PPO in the initial stage of GPO model training to make our comparative analysis thorough.
• Individual Preference Modeling: We will also focus on adapting our approach to model individual preferences. This will involve developing innovative architectures and creating new datasets from user interactions to better understand and align to individual needs.