A Benchmark and Field Study on the Use of LLMs and RAG for Harmonizing Job Descriptions in the Austrian Federal Administration

Job descriptions play a central role in public-sector personnel management, job grading, and organizational development. However, in many administrations they remain fragmented, manually maintained, and difficult to compare across organizational units, limiting transparency and hindering digital transformation. In the Austrian federal public administration, these challenges are compounded by heterogeneous regulatory frameworks and the lack of structured, interoperable job description repositories. This paper explores the use of Large Language Models (LLMs) combined with Retrieval-Augmented Generation (RAG) to support the semi-automatic generation and harmonization of structured job description components. We propose a RAG-based pipeline that generates goals, tasks, activities, and requirements grounded in existing administrative documents and expert-authored descriptions. The approach is evaluated through two studies. First, we perform an automatic summarization and similarity analysis comparing multiple embedding-based, lexical, and n-gram-based methods across job description components. Second, we conduct an expert-based evaluation assessing linguistic quality, correctness, and usefulness of the generated outputs. Results indicate that no single similarity measure performs best across all content types. SBERT-based embeddings achieve the most consistent performance, reaching 96\% rank-based classification accuracy for task descriptions, while expert assessments confirm high linguistic quality but exhibit limitations for more abstract text elements.