PhD on Developing a User-centric AI evaluation framework for LLMs

Position PhD-student

Irène Curie Fellowship No

Department(s) Industrial Engineering and Innovation Sciences

FTE 1,0

Date off 01/12/2024

Reference number V39.7845

Job description

Do you want to develop methods to better evaluate, understand and improve human-AI interaction in the context of value-aligned Large Language models? Do you want to  conduct impactful user research in close collaboration with scientists and engineers in a EU-funded doctoral network, helping to promote fair and transparent generative AI?

We are seeking a highly motivated doctoral candidate to join our research team as part of a prestigious Marie Skłodowska-Curie Actions (MSCA) doctoral network: AlignAI. The AlignAI doctoral network aims to train doctoral candidates to develop, evaluate, and engage with Large Language Models (LLMs). It focuses on aligning these models with human values to ensure their development and deployment are ethically sound and socially beneficial. By integrating expertise from social sciences, humanities, and technical disciplines, the project will address critical issues such as explainability and fairness, thereby ensuring LLMs contribute positively to education, mental health, and news consumption.

The doctoral candidate in this position will develop and test a user-centric AI evaluation framework that triangulates subjective metrics such as perceived transparency, understandability, fairness and cognitive explainability (mental models), with objective measures of explanation and model quality. Quantitative and qualitative user studies will be developed iteratively throughout the project within the three use cases of the AlignAI consortium: Education, positive health and online news consumption. The candidate will support the validation of AI tools developed by other candidates within alignAI consortium and final validation step with user groups towards the end of the project.

The doctoral candidate will be supervised by associate Professor Martijn Willemsen  and embedded within the Human-Technology Interaction group, Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology (TU/e). Additional supervision will be provided within the AlignAI doctoral network by Assistant Professor Sneha Das (Computer Science, DTU) and Professor Daniel Gatica- Perez (Digital Humanities,EPFL). The candidate will be affiliated with the Eindhoven Artificial Intelligent Systems Institute (EAISI), and will be expected to contribute actively and positively to Human-AI interaction research communities in Eindhoven, as well as to attend workshops, summer schools, and research visits (secondments) at AlignAI partner institutions as part of the MSCA doctoral network.

Key Responsibilities

• Develop and test a user-centric AI evaluation framework under supervision of Dr. Martijn Willemsen
• Collaborate with other doctoral candidates in the consortium on validation of the framework by means of several user studies performed within the three use cases
• Engage in secondments at DTU and EPFL as part of the project’s training activities.
• Contribute to academic publications, project reports, and other dissemination activities.
• Participate in structured training programs and network-wide events organized by the consortium.
• Collaborate with international partners and other Doctoral Candidates within the network.

This position is one of four PhD positions being offered at TU/e through the AlignAI doctoral network, the others being in Philosophy & Ethics (Carlos Zednik) and Industrial Design (Jesse Benjamin & Stephan Wensween).

Job requirements

• A master’s degree (or an equivalent university degree) in Human-computer Interaction, Human-Technology Interaction, Computer Science, Cognitive Science, Artificial Intelligence or related disciplines
• Candidate is expected to have a multidisciplinary background that covers user aspects / psychology, research methods and engineering skills in the field of AI
• A research-oriented attitude.
• Applicants must not own a doctoral degree at the date of recruitment.
• Applicants may be of any nationality, but must not have resided or carried out their main activity (work, studies, etc.) in the Netherlands for more than 12 months in the 36 months immediately before the recruitment date.
• Ability to work in an interdisciplinary team and interested in collaborating with industrial partners.
• Motivated to develop your teaching skills and coach students.
• Fluent in spoken and written English (C1 level).

Selection Process

• Application Submission: Applications must be submitted by December 1, 2024.
• Selection Criteria: Candidates will be evaluated based on their academic background, research experience, and alignment with the project’s objectives. The selection process will be open, transparent, merit-based, impartial, and equitable.
• Interview: Shortlisted candidates will be invited for a structured interview with predefined questions and a scoring system.
• Reserve List: A reserve list of top candidates may be created for future opportunities.
• Selection Committee: The selection committee will comprise diverse experts ensuring gender balance, nationality diversity, and relevant experience.

Conditions of employment

A meaningful job in a dynamic and ambitious university, in an interdisciplinary setting and within an international network. You will work on a beautiful, green campus within walking distance of the central train station. In addition, we offer you:

• Full-time employment for four years, with an intermediate evaluation (go/no-go) after nine months. You will spend 10% of your employment on teaching tasks.
• Salary and benefits (such as a pension scheme, paid pregnancy and maternity leave, partially paid parental leave) in accordance with the Collective Labour Agreement for Dutch Universities, scale P (min. €2,872 max. €3,670).
• A year-end bonus of 8.3% and annual vacation pay of 8%.
• High-quality training programs and other support to grow into a self-aware, autonomous scientific researcher. At TU/e we challenge you to take charge of your own learning process.
• An excellent technical infrastructure, on-campus children's day care and sports facilities.
• An allowance for commuting, working from home and internet costs.
• A Staff Immigration Team and a tax compensation scheme (the 30% facility) for international candidates

Information and application

About us

Eindhoven University of Technology is an internationally top-ranking university in the Netherlands that combines scientific curiosity with a hands-on attitude. Our spirit of collaboration translates into an open culture and a top-five position in collaborating with advanced industries. Fundamental knowledge enables us to design solutions for the highly complex problems of today and tomorrow. 

Information

Do you recognize yourself in this profile and would you like to know more?
Please contact the hiring manager Martijn Willemsen, M.C.Willemsen@tue.nl

Visit our website for more information about the application process or the conditions of employment. You can also contact hrservices.ieis@tue.nl or +31 40 247 5152.

Are you inspired and would like to know more about working at TU/e? Please visit our career page.

Application

We invite you to submit a complete application by using the apply button. The application should include a:

• Cover letter in which you describe your motivation and qualifications for the position.
• Curriculum vitae, including a list of your publications and the contact information of three references.
• A writing sample (publication, MSc. Thesis, term paper) that demonstrates your background and writing skills in the area of the PhD project

We look forward to receiving your application and will screen it as soon as possible. The vacancy will remain open until the position is filled.