Applying the Delphi method to determine best practices for outsourcing logistics in disaster relief

Purpose – The purpose of this paper is to determine best practices of aid agencies for outsourcing logistics to commercial logistics service providers (LSPs) in disaster relief. Moreover, it evaluates the application of the Delphi method for research in humanitarian logistics. Design/methodology/approach – The paper is based on a two-round Delphi study with 31 experts from aid agencies and a complementary full-day focus group with 12 experts from aid agencies and LSPs. Findings – The study revealed 12 best practices for outsourcing logistics in disaster relief and a compilation of more than 100 activities for putting these practices into action. Experts consider a proper balance between efficiency and compliance, a detailed contract and a detailed service request most important. Additionally, the Delphi method was found to be a promising technique for research on humanitarian logistics. Research limitations/implications – By critically examining the Delphi method, this study establishes the basis for a wider application of the technique in the field of humanitarian logistics. Furthermore, it can help to prioritize future research as the ranking of practices reflects the priorities of practitioners. Practical implications – The paper provides guidance to practitioners at aid agencies in charge of outsourcing logistics. Originality/value – This research is one of the first in the field of humanitarian logistics to apply the Delphi method. Moreover, it addresses the lack of literature dealing with approaches for building successful cross-sectoral partnerships.


Introduction
Outsourcing logistics in disaster relief operations provides considerable benefits to aid agencies, such as cost efficiency, flexibility and scalability (Baharmand et al., 2017;Cozzolino et al., 2017;Thomas and Fritz, 2006).Therefore, many aid agencies make strong use of outsourcing, and multiple billion US$ are spent every year by aid agencies on logistics services (Binder and Witte, 2007;L'Hermitte et al., 2016).Commercial logistics service providers (LSPs) are an integral part of any disaster relief operation, both at the international and the local level (Sánchez Gil and McNeil, 2015;Vega and Roussat, 2015).
However, many organizations fail to build successful collaborative relationships with LSPs (Bealt et al., 2016).Often, they are dissatisfied both with the perceived performance and the related costs (Cottam et al., 2004;Schulz, 2009).The success of outsourcing engagements is affected by specific challenges of the humanitarian environment.For example, aid agencies' high staff turnover and unpredictable funding cause issues for building relationships with LSPs (Thomas, 2003;Van Wassenhove, 2006).Likewise, different cultures and working styles cause frictions during the collaboration (Nurmala et al., 2017).In the light of these challenges, aid agencies seek guidance on how to best leverage the advantages of outsourcing while reducing the risks and downsides of such engagements.Their scope of action is wide in this regard, covering activities such as provider selection, contract negotiation, performance evaluation or supplier management (Bagchi and Virum, 1996;Gould, 2003;Sink and Langley, 1997).Some approaches in these areas, called best practices, consistently show results superior to those achieved by other approaches (Business Dictionary, 2018;Supply Chain Council, 2008).Taking a more strategic and structured approach to outsourcing by replicating such best practices can help aid agencies to be more successful in outsourcing engagements (Bealt et al., 2016).Therefore, in response to aid agencies' need for guidance, we address the following research question: RQ.What are best practices of aid agencies for ensuring the success of outsourcing logistics to commercial LSPs in disaster relief?
Since academic literature in this regard is very sparse and did not address the same level of detail as our study (Bealt et al., 2016;Nurmala et al., 2017;Vega and Roussat, 2015), answering our research question required an exploratory approach.Moreover, the environment of disaster relief is well known for its lack of performance data and indicators (Abidi et al., 2014;Beamon and Balcik, 2008;Schulz and Heigh, 2009).Therefore, we decided to leverage the knowledge of humanitarian practitioners who have personal experience in engaging LSPs and, consequently, expert knowledge about advisable practices.As the responses of individuals are necessarily judgmental and situational, individual opinions need to be combined in order to develop a comprehensive picture.The Delphi method is particularly suited for exploratory research under such conditions (Okoli and Pawlowski, 2004;Akkermans et al., 2003;Meredith, 1993;Turoff, 1970).It is an anonymous and iterative technique for facilitating and structuring communication among a group of experts with the objective of either transforming individual opinions into group consensus or identifying systematic dissent among participants (Dalkey and Helmer, 1963;Hasson et al., 2000;Rowe and Wright, 2011).While common surveys are best at investigating "what is", Delphi studies excel in asking "what […] should be" (Hsu and Sandford, 2007).Consequently, it is very appropriate for our type of research question and has already been leveraged in this context in other disciplines (e.g.Ager et al., 2010).
The Delphi method brings a number of further advantages for research in humanitarian logistics.First, it allows to account for the worldwide dispersion of humanitarian experts, since it can be conducted remotely and asynchronously.Second, the method does not require the expert panel to be statistically representative of any population, because the expertise of the panelists is more important than their number (Ludwig, 1997;Okoli and Pawlowski, 2004;Powell, 2003).This makes the approach more feasible in the humanitarian environment than large-scale surveys with representative samples, which are prone to struggle with the non-transparent group of existing aid agencies (Vega and Roussat, 2016).Third, the Delphi approach guarantees anonymity of responses (McKenna, 1994) and encourages humanitarian experts to express their true opinions.Given the competition between aid agencies and the importance of each organization's reputation, practitioners might be reluctant to do so in case of non-anonymous research methods.Fourth, the Delphi method resolves group interaction effects such as the dominance of powerful individuals (Dalkey et al., 1972;Goodman, 1987; 439 Outsourcing logistics in disaster relief Rowe et al., 1991) and prevents the results from being biased toward humanitarian opinion-leaders, for instance, major aid agencies or UN organizations.
Despite the described benefits, only a few studies in humanitarian logistics have applied the Delphi method to date (Cottam et al., 2004;Mari Ainikki Anttila, 2014;Richardson et al., 2016;Sahebi et al., 2017).Therefore, as a secondary objective in addition to our content-oriented research question, we critically explore the applicability of the research method and its ontological, epistemological and methodological assumptions.
In summary, our research contribution is twofold.First, we determine a set of best practices, which can help humanitarian organizations to be successful when outsourcing logistics.To provide concrete guidance to humanitarian practitioners, our focus is on the compilation and discussion of detailed activities for implementing the best practices in disaster relief operations.Second, we shed light on the application of the Delphi method as a promising method for empirical research in humanitarian logistics.The critical debate of our research design is intended to support other researchers to successfully leverage the method for their own research.It has to be noted, however, that an all-embracing discussion of the Delphi method would require a designated paper.This paper focuses on the most important aspects in reference to our own study.
This paper is structured as follows.In the next section, we briefly review the relevant literature.In Section 3, we describe our research design and discuss its trustworthiness from a methodological perspective.In Section 4, we present the results of the empirical study, which we then discuss in Section 5 in view of existing literature.We close the paper with a summary and an outlook in Section 6.

Literature review
The existing literature on outsourcing in humanitarian logistics can be grouped into four categories.The first group of papers investigate the benefits, risks, barriers and enablers of cross-sector collaboration and partnerships between aid agencies and LSPs.Representatives of this stream of research are Balcik et al. (2010), Bealt et al. (2016), Cozzolino (2012), Maon et al. (2009) and Nurmala et al. (2017Nurmala et al. ( , 2018)).The second group of studies focus on specific challenges of cross-sector partnerships, such as service contracts (Egan, 2010), cultural barriers (Dowty and Wallace, 2010) or cooperative purchasing initiatives (Pazirandeh and Herlin, 2014).The third group of articles explore the role of LSPs and outsourcing in humanitarian operations.Examples for this class of papers are Abidi et al. (2015), Baharmand et al. (2017), Cottam et al. (2004), Cozzolino et al. (2017), Heaslip (2013), Sánchez Gil and McNeil (2015) and Vega and Roussat (2015).The fourth group of studies are mathematical models which examine the efficiency of outsourcing (Nagurney et al., 2011;Wang et al., 2016), the tendering and selection process (Bagchi et al., 2011;Paul and Wang, 2015;Trestrail et al., 2009), and routing decisions of service providers (Huang et al., 2015).
While none of the previous studies has investigated how aid agencies can best ensure the success of outsourcing logistics, some of them mention relevant recommendations as a by-product of their research.Aid agencies should build close relationships with service providers to enable smooth planning and communication (Cottam et al., 2004).To achieve this, they should establish bi-directional data and information exchange, leveraging information technology and ensuring the protection of sensitive data (Balcik et al., 2010).In the best case, aid agencies should establish long-term strategic partnerships, which do not only involve the sharing of resources, but also the sharing of risks (Maon et al., 2009).Building this type of partnerships requires to understand which type of partnership mechanism applies for which situation (Nurmala et al., 2018) and to employ a high level of commitment, resolution, communication and partner focus (Cottam et al., 2004).Since different cultures and working styles can imply challenges for building partnerships (Nurmala et al., 2017), aid agencies should create internal awareness for cultural differences, for the expectations of the service provider and for the rationale behind both aspects (Dowty and Wallace, 2010).They should also adapt to the commercial sector by allowing more decentralized decision-making (Baharmand et al., 2017) and increasing the standardization of their processes (Bealt et al., 2016).
It can be beneficial for aid agencies to combine corporate resources (Maon et al., 2009) and engage in collaborative purchasing of logistics services (Pazirandeh and Herlin, 2014).However, mutual trust among the partners, a harmonized and coordinated tender process, an integrated approach to contracting, and harmonized agency processes and policies are important prerequisites for collaborative purchasing to be successful (Pazirandeh and Herlin, 2014).Alternatively, it can be advantageous to rely on 3PL or 4PL vendors for consolidating demand and managing the sub-contracting of providers (Cottam et al., 2004;Abidi et al., 2015).Aid agencies need to be clear and transparent on their requirements (Bealt et al., 2016) and consider a provider's proven track record, existing relationship network, local knowledge and territorial coverage (Baharmand et al., 2017).They should not rely on single providers, but pursue multiple sourcing (Pazirandeh and Herlin, 2014) and build a network of partnerships (Nurmala et al., 2018).Finally, humanitarian organizations should ensure to have formal service contracts in place, which in the best case are set up in advance of a disaster (Baharmand et al., 2017), transfer the responsibility for the safety of goods to the provider (Baharmand et al., 2017) and are in line with local legislation and practices (Cottam et al., 2004).
In summary, the related literature on humanitarian logistics mentions a number of promising activities for aid agencies engaged in outsourcing, but provides a neither comprehensive nor integrated picture.The relevant literature on commercial logistics, in turn, gives a broader perspective, but describes only high-level factors, which are too abstract for being implemented by aid agencies.For example, the corresponding papers do not explain how relationships with mutual trust and understanding can be built or how a strong performance orientation can be established.Moreover, the insights from commercial logistics cannot be applied to the humanitarian context without adaptations because of the 441 Outsourcing logistics in disaster relief sector's well-documented specifics (Kovács et al., 2009;Van Wassenhove, 2006).Accordingly, multiple researchers point out a need for dedicated research, especially empirical studies, on how aid agencies and LSPs can build successful outsourcing relationships (Bealt et al., 2016;Nurmala et al., 2017;Vega and Roussat, 2015).We address this gap with respect to both content and methodology by determining best practices for outsourcing logistics in disaster relief based on a Delphi approach and by compiling and discussing detailed activities for implementing these practices.

Research design
We conducted a two-round Delphi study between November 2017 and March 2018 with 31 experts from humanitarian organizations.It was administered electronically using the Calibrum Surveylet Software (Aengenheyster et al., 2017) and can, consequently, be called e-Delphi (Hasson and Keeney, 2011).After the finalization of the Delphi study, in April 2018, we conducted a complementary full-day focus group with ten experts from humanitarian organizations and two experts from LSPs (Morgan and Krueger, 1997;Morgan, 1996;Kitzinger, 1995).The purpose of the focus group was to perform a member check on the results, that is, to validate the outcome of the e-Delphi as well as our analysis and interpretation with subject matter experts (Creswell, 2009;Wallendorf and Belk, 1989;Lincoln and Guba, 1985).

The e-Delphi
The four basic principles of the Delphi method are an iterative and multistage process, some level of controlled feedback, the opportunity for individuals to revise their answers and a certain degree of anonymity (Linstone and Turoff, 1975;Hasson et al., 2000;Powell, 2003;Hsu and Sandford, 2007).While all Delphi studies share these common characteristics, the flexibility of the Delphi method has led to a high diversity of methodological variants (Hasson and Keeney, 2011).One important variant of Delphi design is the Policy Delphi, which differs considerably from the classical Delphi (Turoff, 1970).Its prime objective is not to seek consensus for decision-making or forecasting under uncertainty, but to identify and evaluate potential solutions for a selected issue and to reveal arguments for and against each solution (Turoff, 1970).Accordingly, its approach is rather qualitative, even if quantitative questions might be applied to evaluate the different alternatives.Determining best practices for outsourcing logistics in disaster relief requires to identify and evaluate different ways of managing activities such as provider selection, contract negotiation, performance evaluation or supplier management.We have, therefore, built our research design on the principles of the Policy Delphi.
Number of rounds.Delphi results tend to show only slight or even no improvements after the second iteration (Woudenberg, 1991).At the same time, the risk of biased results increases with the number of conducted iterations because of participant fatigue and drop-outs (Hasson et al., 2000).Accordingly, most Delphi studies consist of two or three rounds (Diamond et al., 2014).It is generally advisable to limit the Delphi study to the minimally sufficient extent and verify its results through follow-up explorations (Delbecq et al., 1975).We had already achieved a high level of consensus after the second round and perceived participant fatigue to become a relevant risk.Additionally, given our research objective, the explanations of panelists in reference to their second round judgments were considerably more important for our findings than minor changes of quantitative votes in a potential third round.Therefore, we limited the study to two rounds, knowing that any need for clarification could be addressed in the complementary focus group.
Expert panel.While there are no definite criteria for the definition and selection of experts (Hasson et al., 2000;McKenna, 1994;Hsu and Sandford, 2007), it is clear that the expertise and experience of the panelists determine the trustworthiness of the Delphi results

442
JHLSCM 9,3 (Hasson and Keeney, 2011;Hsu and Sandford, 2007;Lincoln and Guba, 1985).On a general level, experts are persons working in the relevant field with substantial knowledge of the topic under investigation (McKenna, 1994;Powell, 2003).For our study, we defined an expert as a humanitarian practitioner who had considerable and personal experience in outsourcing of logistics in disaster relief.Such persons possess access to privileged information which is difficult for other people to obtain and process.Accordingly, they are in a position to structure the field of action in a meaningful way and advise others on what to do (Bogner et al., 2014;Bogner and Menz, 2009;Meuser andNagel, 1991, 2009).We requested potential participants to only register for the study if they possessed the described type and level of experience.In addition, we validated their expertise in Round 1, asking them to describe their personal experience with outsourcing along a number of dimensions.If their descriptions were coherently elaborated, their experience was seen as a sufficient basis for sharing information and providing advice to others (Stehr, 1994).
Most Delphi studies involve between 11 and 50 experts (Diamond et al., 2014;Hsu and Sandford, 2007).Since our research interest was to explore the perspective of aid agencies, we decided to not include any representatives of LSPs in the expert panel.For recruiting experts from aid agencies, we applied a mix of purposive sampling techniques: criterionbased sampling, snowball sampling and opportunity sampling (Patton, 2002).On the one hand, we approached all organizations which were listed as most important disaster relief agencies by Vega and Roussat (2016) based on the report Global Geneva's 2015 Top 500 NGOs.Through either personal contacts or central mailboxes and phone lines we asked these organizations for employees who had personal experience with engaging LSPs in disaster relief operations and were willing to participate in our study.On the other hand, we identified relevant experts by asking our existing personal contacts at humanitarian organizations for further qualified participants within their network.In total, we recruited 31 experts from 24 humanitarian organizations who had on average 12 years of experience in humanitarian logistics, both at headquarters and field level (see Tables AI and AII).Their high level of experience, their qualifications and their long-lasting exposure to the problem under investigation contributed to the trustworthiness of our results (Hasson and Keeney, 2011;Lincoln and Guba, 1985).
Round 1.All recruited experts participated in the first round of the study, which was a qualitative online survey [1].Before releasing the questionnaire to the panel, we pre-tested it with selected practitioners.The survey consisted of three parts (see Appendix 3): first, panelists were asked to describe a situation in which they had worked personally with LSPs, focusing on what they would do either similarly or differently under comparable conditions in the future.This framed the research subject and stimulated the link to the panelists' experiences (Froschauer and Lueger, 2003).Second, they were requested to list success factors for outsourcing logistics in disaster relief in general.Third, they were prompted to provide specific recommendations with respect to the areas of provider selection, contract negotiation, performance evaluation and supplier management.
We analyzed the responses of the first survey, that is, the textual data, based on the principles of content analysis in NVivo.While NVivo, a leading tool for qualitative data analysis, was used to organize, store and retrieve the data, the actual analysis was carried out manually by the research team.Based on rules of procedure we assigned conceptual labels to the empirical data to explicate, summarize and structure the recommendations of participants (Mayring, 1991(Mayring, , 2000)).The coding process combined both inductive and deductive approaches (Fereday and Muir-Cochrane, 2006).During deductive coding we assigned labels which were suggested by existing literature (theory-driven coding).When the suggestions of experts could not be referenced to existing literature, we applied inductive coding, that is, we assigned labels which were suggested by the data itself

443
Outsourcing logistics in disaster relief (data-driven coding).The coding process was organized as an iterative procedure in order to deal with inter-observer variability.First, three researchers with different professional backgrounds individually grouped the activities.Afterwards, multiple meetings were held to compare, discuss and consolidate the different classifications until there was perfect agreement among the researchers.
The analysis of the first round revealed a set of 113 activities which were recommended for ensuring the success of outsourcing by at least one expert.These activities were grouped into 12 higher-level categories, so-called practices, based on commonalities with respect to the objective of each activity.
Round 2. For validating the findings from the first round, we fed the anonymized results of our analysis back to the expert panel as part of a rather quantitative second online survey (see Appendix 4).This mix of qualitative and quantitative methods contributed to the methodological rigor of our study (Hasson and Keeney, 2011).As for the first round, we pre-tested the questionnaire before releasing it to the panel.In total, 27 experts participated in this survey (87 percent response rate).They rated both the importance of the 12 practices and the usefulness of the 113 corresponding activities on a four-point Likert scale.By limiting each page of the survey to a maximum of ten questions, we ensured that the response process was very manageable for participants.The design of both Likert scales was based on the recommendations by Turoff (1970).They featured an optimum number of alternatives, ensured consistent cognitive distances between adjacent pairs of points and prompted experts to take a clear position (Leung, 2011;Lozano et al., 2008).In addition to their quantitative rating, experts had the possibility to explain the reason of their judgment in form of an open comment (Singer and Couper, 2017).
Analyzing the replies of the second round, we calculated the mean rating of all items (activities and practices) and determined the level of consensus among the 27 panelists.We identified items lacking consensus by combining two well-established indicators (von der Gracht, 2012): interquartile range (IQR) and r wg .IQR measures the difference between the lower and upper quartiles.Following the recommendations from Raskin (1994) and Rayens and Hahn (2000), we marked all items with an IQR greater than 1 as "lacking consensus."r wg is a measure of interrater agreement ( James et al., 1993).In line with LeBreton and Senter (2008), we also labeled all items with r wg ⩽ 0.30 as "lacking consensus".In addition to the quantitative analysis, we applied content analysis to the participants' qualitative explanations, following the same procedure as in Round 1.
Our analysis revealed a high level of consensus among experts; panelists had expressed controversial views for only 24 activities and 1 practice (see Section 4).There was a considerable level of doubt if a third round could resolve the remaining controversies (Woudenberg, 1991), especially because participant fatigue had reached a potentially risky level (Hasson et al., 2000).We, therefore, decided to terminate the e-Delphi after the second round and to use follow-up explorations for shedding more light on the remaining controversies (Delbecq et al., 1975).The complementary face-to-face focus group was used for this purpose.

The focus group
The focus group (Morgan and Krueger, 1997;Morgan, 1996;Kitzinger, 1995) was used as a member check (Creswell, 2009) and aimed to validate the outcome of the e-Delphi as well as our analysis and interpretation with subject matter experts.Member checks are a crucial technique for establishing credible and trustworthy findings (Lincoln and Guba, 1985).They are an opportunity to reveal latent assumptions of experts, challenge interpretations of researchers and obtain additional information from participants which may be stimulated by the playing back process.Accordingly, member checks should be treated as another 444 JHLSCM 9,3 source of data and insights (Silverman (2013) in reference to Fielding and Fielding (1986)).Used in this way, the focus group allowed us to gather additional textual data to enrich the findings from the e-Delphi.Additionally, it enabled us to understand in detail the reasons for the dissent among panelists regarding 24 activities and 1 practice.Debating both benefits and drawbacks of the particular items in a controversial, face-to-face discussion with selected experts allowed us to take a deep dive into the underlying assumptions and contextual relationships (Rowe and Wright, 2011;van de Linde and van der Duin, 2011).
In order to allow fruitful discussions within a diverse group of experts, we limited the number of focus group participants to ten and restricted participation to one representative per organization.Given the administrative and financial challenge of bringing together an internationally distributed group of experts for a face-to-face meeting, we recruited the focus group participants based on opportunity sampling (Patton, 2002).On the one hand, we invited all respondents of the second survey to the focus group.On the other hand, in order to account for the busy schedule of humanitarian practitioners, we offered panelists the opportunity to send equally qualified substitutes from their organizations in case they could not attend themselves.Focus group seats were awarded according to the sequence of registration.To ensure a representative pooling of judgments and experiences (Okoli and Pawlowski, 2004;Powell, 2003;Ludwig, 1997), we carefully monitored the process of participant registration, prepared to correct any potential bias in the sample through purposive sampling (Patton, 2002).In the end, a very competent and diverse group of experts was recruited for the focus group, consisting of five respondents of the second survey and five nominated substitutes (see Table AI).The new participants allowed us to reflect on the findings from an outside perspective and to incorporate new viewpoints into the results.In addition to the experts from aid agencies, we invited two representatives of commercial LSPs to the focus group in order to mirror the perspective of aid agencies.The presence of LSPs helped us to trigger truly controversial discussions.Through such discussions we were able to gain insights into, by then, tacit knowledge and assumptions of experts, which had not been accessible by the previous question-and-answer surveys (Rowe and Wright, 2011;van de Linde and van der Duin, 2011).
The full-day focus group consisted of multiple sessions.In each session, the research team first presented results from the two online surveys.In order to direct the participants' attention to the content, methodological details were not explained.Afterwards, discussions took place among the experts.A member of the research team facilitated the discussions, summarized them and asked participants for validation or corrections.The focus group was audio-recorded, transcribed word-for-word and, afterwards, content-analyzed in NVivo, following the same procedures as for the textual data from the e-Delphi.

Methodological discussion
Even though the Delphi method entails considerable benefits, it is also subject to constant critique (Hasson and Keeney, 2011).Therefore, we deem it important to discuss the trustworthiness of our research design.Due to the qualitative and post-positivist nature of our study and its underpinning ontological and epistemological assumptions, the concepts of validity, reliability and objectivity are not applicable to our work (Day and Bobeva, 2005;Wallendorf and Belk, 1989).Instead, we will discuss the criteria of credibility, transferability, dependability and confirmability (Lincoln and Guba, 1985), which are well established for qualitative research.In doing so, we will focus on challenges which future Delphi studies in humanitarian logistics are likely to encounter and potential solutions to these challenges.
Credibility.Discussing our results in the face-to-face focus group with a group of experts, partially internal and partially external to the Delphi, contributed to the credibility of our study in form of a member check (Wallendorf and Belk, 1989).This allowed us to take a deep 445 Outsourcing logistics in disaster relief dive into the panelists' underpinning assumptions and logics and to refine the results accordingly (van de Linde and van der Duin, 2011).While this was an important part of our research, this might not be possible in other studies on humanitarian logistics due to reasons of time and cost.In the interest of credibility, we also applied triangulation across researchers and sources, for example, by performing content analysis within a heterogeneous team and by reviewing the results in the context of existing literature (Wallendorf and Belk, 1989).Still, incorporating further documents into the analysis, such as actual logistics contracts, could have been a promising extension of the triangulation approach.
Reflecting critically, it should also be noted that our study was executed in English.Since most of the participants were not native speakers, lingual misconceptions might have influenced the results.In general, misconceptions might also have occurred because of formulations in the questionnaires.For example, participants might have been misled by the perception of overlaps in the titles of the 12 practices.We tried to counteract by using simple expressions and unambiguous wording (Frewer et al., 2011;Parente and Anderson-Parente, 2011) and by performing pre-tests with selected practitioners for both surveys (Okoli and Pawlowski, 2004).The pre-tests also helped us to ensure that participants were able to comfortably handle the design, length and complexity of the questionnaires.While we perceived this approach beneficial, further measures could have been applied, for example, debriefing by peers during the development and analysis of the questionnaires (Lincoln and Guba, 1985).
Another limitation to credibility, which is inherent to all Delphi studies, is the influence of the selected consensus measures and the corresponding decision to terminate the study.A vast abundance of approaches exists and different measures might lead to different conclusions regarding the level of consensus (Diamond et al., 2014;von der Gracht, 2012).Our focus was not the achievement of consensus on a quantitative measure, but the identification of a set of important practices.Given this rather qualitative approach, the explanations of panelists in reference to their judgments were considerably more important for our findings than the choice of the consensus measure and the revision of the second round votes in a third round.For other applications in the field of humanitarian logistics, however, we recommend a critical consideration of this aspect.In conclusion, we have a good level of confidence in the credibility of our research.However, "as with most research, findings are never completely definitive but can be strengthened, clarified and enhanced by judicious and sound follow-up research" (Kennedy, 2004).
Transferability.We established a heterogeneous expert panel by recruiting participants from diverse organizations and with a broad range of different experiences in disaster relief.Such triangulation through purposive sampling helped to enhance the transferability of results (Delbecq et al., 1975;Linstone and Turoff, 1975;Wallendorf and Belk, 1989).However, we faced limitations in this regard when trying to recruit experts from African or Asian countries, which were due to a lack of accessibility (for the Delphi) and high cost (for the focus group).This might have biased the results toward the perspective of European experts, even though our panel had extensive experience in the field.Future studies should put an extra eye on this topic and identify specific approaches for involving non-European panelists.As another measure for improving transferability, we sought for limiting exceptions throughout our study, which could define the boundaries of our results (Wallendorf and Belk, 1989).For this reason, we asked panelists to justify their judgments and provided them the opportunity to outline the reasoning behind their responses (Nowack et al., 2011).Panelists brought up some relevant boundaries for the transferability of our results on these occasions.For instance, the usefulness of selected activities could be influenced by contextual factors, such as the value of the service purchase, size and funding structure of an aid agency, the geographic context of the service (before or after point-of-entry) or the phase of disaster relief.However, such exceptions do not imply that 446 JHLSCM 9,3 our results need to be modified (Glaser and Strauss, 2017;Wallendorf and Belk, 1989).Even when exceptions represent a considerable number, there can be substantial evidence for the acceptance of the results (Lincoln and Guba, 1985).Reflecting all contextual factors of humanitarian logistics in the results would be nearly impossible.Instead, it is important to recognize that the application of the best practices by aid agencies might be subject to slight variations, depending on the contexts.Future Delphi studies could account for this challenge by working with multiple parallel panels of experts, which ensure homogeneity of each panel with respect to the important contextual factors.
Dependability.Authors such as Cornick (2006) are confident that the use of a diverse and qualified expert panel would secure the stability and dependability of Delphi results.While we have designed our panel in exactly this way, we still consider it important to recognize the limitations of the Delphi method in this regard.According to the post-positivist paradigm, a single, objective reality does not exist (Lincoln and Guba, 1985).Both people and contexts change continually (Wallendorf and Belk, 1989).Accordingly, there is no guarantee that our results would repeat if we replicated the study with similar persons in a similar context.Therefore, Wallendorf and Belk (1989) recommend to explore the boundaries of dependability by applying a longitudinal approach and repeating investigations months or years later when things should have changed in various ways.By investigating these changes, researchers would be able to assess if the original theory is still applicable or needs to be modified.Given the dynamics of the humanitarian sector, different results in multiple years from now should actually not be surprising.Although this might appear a considerable limitation, this challenge is inherent to all qualitative research approaches.Furthermore, one needs to bear in mind that the Delphi method does not strive to generalize, but aims to provide a snapshot of expert opinion at a specific moment in time (Maceviciute and Wilson, 2009;Thompson, 2009).If researchers take this into account for their interpretation, and combine it with a longitudinal design as far as financial resources allow, dependability should not be a barrier for a wider use of the Delphi method in research on disaster relief.
Confirmability.Post-positivists philosophy also qualifies the requirement of confirmability: "There can be no absolute objectivity; at best the researcher can become conscious of and hopefully reduce […] biases" (Wallendorf and Belk, 1989).Therefore, a clear audit trail of data gathering and interpretation is one of the most important measures for enhancing confirmability in Delphi research (Rodgers and Cowles, 1993;Skulmoski et al., 2007).We established this by transcribing all interviews and focus group discussions, storing all survey responses in a database and recording notes on our analysis in NVivo.Also, to increase confirmability, we leveraged triangulation across researchers by involving multiple researchers in the planning and execution of interviews and focus groups as well as the analysis of textual data (Wallendorf and Belk, 1989).While we consider these techniques appropriate, future studies based on the Delphi method could, nevertheless, apply further measures in this regard.For example, they could make use of external auditors which are supplied with all raw materials as well as their interpretation (Wallendorf and Belk, 1989).Moreover, they could formally determine inter-coder reliability in order to assess the confirmability of the coding process (Mayring, 2000).

Results
In the first survey, 31 experts recommended 113 activities for ensuring the success of outsourcing logistics.As described in detail in Section 3, these were grouped into 12 best practices and presented for further evaluation in the second round.In that survey, 27 panelists rated the importance of each practice on a four-point Likert scale (not important, somewhat important, important and very important).Table I shows the mean rating of each practice.Three practices were rated very important (mean rating ⩾ 3.50) and nine practices 447 Outsourcing logistics in disaster relief important (2.50 ⩽ mean rating o 3.50).In general, there was consent among panelists (IQR ⩽ 1 and r wg W 0.30).Only for the importance of practice P07, no consent was achieved.
In total, 27 panelists also rated the usefulness of each activity on a four-point Likert scale (not useful, somewhat useful, useful and very useful).Table AIII shows the mean rating of each activity.While 35 activities were considered very useful (mean rating ⩾ 3.50) and 71 useful (2.50 ⩽ mean rating o 3.50), 6 activities were found to be only somewhat useful (1.50 ⩽ mean rating o 2.50) and 1 was rated not useful at all (mean rating o 1.50).For the majority of activities (89 activities), experts agreed on their usefulness (IQR ⩽ 1 and r wg W 0.30).For 24 activities, some level of controversy existed among panelists.In some cases, experts did not agree if an activity was useful or very useful.In other cases, experts had opposing views.While one group considered an activity very useful, another group deemed it not useful at all.These items were discussed in detail during the focus group in order to identify the reasons for this controversy.
We will now describe in detail the 113 activities (Table AIII) which were suggested by the panel for implementing the best practices (Table I).We will use expert quotes from the e-Delphi and the focus group to exemplify the recommendations.Where applicable, we will outline the reasons for existing controversies.
4.1 Rank 1: ensure both compliance and efficiency in the procurement process (P03) The most important practice, according to the experts, is to ensure both compliance and efficiency in the procurement process and to properly balance both.Aid agencies face two requirements, which are often conflicting: speed and compliance.On the one hand, their "main objective is to save lives" (E19)[2] and any delays or inefficiencies in the procurement process put this target at risk.On the other hand, many humanitarian organizations "are donor driven agencies [and] have to, even in the emergency, maintain a level of compliance" (E25) with donor requirements, for example, a fair and transparent, but also time-consuming tender process.
Many times, "the good selection, which […] fulfil[s] all the necessary accountab[ibility] procedures, is in direct opposition to the tight and quick deliveries" (E35).Therefore, experts recommend to align procurement procedures to donor requirements and public procurement regulations, but to allow derogations from these procedures in the case of "sudden-onset scenarios or where [a] beneficiary's life is at risk" (E21).For example, long-term agreements should be tendered openly and publicly and be awarded based on the decision of a multi-person tender committee.Short-term contracts in the context of emergencies, however, can be awarded either based on restricted tenders, where a request for proposal is sent to a limited number of suppliers only, or even without competitive bidding based on direct negotiations with selected providers.This is "is quite often by many donors accepted [sic.]"(E35) since they "understand the concept of [a] life-saving emergency" (E25).The challenging part is "to determine when the emergency stops, and at which point the organization has to be ready to roll-out the proper [procedures]" (E25).The danger "is that if you start […] an operation based on derogations, there is a tendency that the organization continues to use derogations, and derogations, and derogations, and never takes that next step to operate in the correct manner" (E5).For example, "in the Syria response […] [t]here were agencies that were keeping emergency response for two years" (E25).Derogation rules can be abused by people in situations of self-imposed urgency, which are due to internal shortcomings and not the disaster itself.Therefore, experts recommend to strictly regulate the use of derogation rules by defining appropriate approval workflows, for example, "senior managers to look into the urgency of the request to determine whether there is a justification" (E25).Moreover, top management should clearly define the duration of the emergency status, potentially also based on external reports.One agency, for instance, only allows exceptions due to emergencies if the particular location at the given time is formally recognized by the European Commission as a crisis situation.Besides defining clear exception rules, organizations should carefully document every purchase from request to decision, also in emergencies.Finally, experts emphasize that, given appropriate preparation and planning (see Sections 4.4 and 4.5), compliant procurement is also possible under time pressure: "I think in 99% of our businesses you can tender this stuff " (E6).
4.2 Rank 2: use a detailed written contract as safeguard (P06) Contracts serve as safeguards for aid agencies by documenting the agreement with the LSP and providing the legal ground for enforcing the agencies' claims in case of any opportunistic behavior of the provider.Furthermore, organizations can use contracts as a manual of cooperation and to establish "a very clear understanding and description of the mutual expectations" (E23).Accordingly, experts deem it very important to set up a detailed and written contract: "Contracts […] with logistics service providers, especially in the beginning of […] operations, must be very carefully written" (E2), because "what is not written, is not in the contract" (E19).
According to the experts, the contract should explain the context of the operation and the specifics of a humanitarian mission, such as urgency or uncertainty.It should specify the exact scope and duration of the service, clarifying "the differentiation between included and not included" (E27) activities to forestall any discussions about gray areas of responsibility.In addition, agencies should outline the service provider's supplementary responsibilities, such as documentation or reporting, and describe guidelines for the collaboration, such as planning or communication procedures.Attaching a code-of-conduct, data protection rules or non-disclosure agreements are considered further useful measures in this regard.
The contract should also include performance indicators and service-level agreements.Only in this way, agencies can "assess […] providers' service level and performance against contractual agreements" (E24) (see Section 4.8).These should be complemented by specifying the consequences of a failure to fulfill the service-level agreements (e.g.penalties or liquidated damages).Likewise, provisions for other adverse events, such as product damages or theft, should be recorded in the contract.Regulating the responsibility of securing insurance coverage is paramount in this regard.
Further important elements of the contract are the structure of service fees and the applicable payment terms.In this context, aid agencies should, for instance, set safeguards against price increases due to unforeseen circumstances: "Price adjustments after finalizing the contract must not happen.The provider has to handle changing conditions" (E18).

Outsourcing logistics in disaster relief
While this approach should be balanced with the objective of creating win-win situations (see Section 4.11), conflicts are occasionally inevitable.Therefore, sufficient provisions for adjusting or terminating the contract are advisable.Likewise, the contract should outline the approach to dispute arbitration, the governing law and the court of jurisdiction."the more info [is] shared, [the] less issues come up at the negotiation and signing stage" (E24).In this regard, it can also help to attach a draft contract with standard terms and conditions to the service request since this "avoids any surprises down the road" (E5).This is, in particular, important for services at the field level; international service providers often prefer to use their own standardized contracts.Experts highlight, furthermore, that is not always "possible to give all exact details upfront" (E21) as the requester might not have all required information either.In these cases, it is important "to give firm and good indications, but [to allow for] some flexibility" (E21).
In addition to detailed requirements, experts recommend to highlight the context of the operation and the specifics of a humanitarian mission in the request to the provider.Moreover, in order to establish transparency, agencies should clearly explain the bidding and selection process (e.g.administrative requirements, timeline and selection criteria).Attaching standardized bid templates and questionnaires for additional information (e.g.customer references) enables a smooth and efficient process for both agencies and service providers.The latter is very important, because "many vendors find the detailed documentation and need for information from them an obstacle" (E28).Extensive bidding requirements can lead to lower response rates from LSPs and hinder the selection of the right provider (see Section 4.6).
4.4 Rank 4: prepare for the engagement of logistics service providers (P01) Panelists consider it important to make sufficient preparations for the engagement of LSPs: "Bringing things into the country […] has to be arranged before the emergency actually occurs" (E25).Preparation can take different forms.Organizations should, first of all, develop guidelines, process documentations and standard operating procedures, which define how the organization handles outsourcing, for example, regarding the procurement of services or the setup of contracts.Especially in emergencies, such documents "help to avoid confusion" (E22) and allow "to be efficient" (E34), because "if there is no documentation, […] it is going to be a big chaos" (E17).
It is even more important, though, that these procedures are actually put into practice and "not merely for documentation" (E31).This is one of the reasons why experts advise organizations to sufficiently train their employees, for instance, with respect to tender processes or vendor management.Establishing a group of experts in central procurement offices can help in this regard.However, it is highly recommended to also sufficiently train field staff which is interacting with LSPs.As Expert 28 explained, "signing contracts for those people who're not good professionally is very, very difficult because they don't understand it.
[…] [E]ven the tender processes are sometimes too complex for them."Regular team meetings for discussing "lessons learnt" are an important addition to formal training sessions.
Personnel development should be complemented by developing appropriate tools and templates, for example, standardized contracts, checklists for performance evaluations or provider databases.The latter should be fed by regular assessments of service provider 450 JHLSCM 9,3 markets, in which new potential providers are identified, market prices are compared and cost structures of providers are analyzed.Expert 8 underlined their importance: "We reduced our cost […] by one million US Dollar over the first year after we started actually doing market assessments."At best, organizations do not only assess service provider markets, but also run pre-qualification procedures and set up framework agreements covering "as many eventualities, situations [and] supply chain requirements [as possible]" (E21).These long-term contracts usually do not contain any volume commitments and sometimes not even fixed prices.Still, "longer-term framework contracts [are] a very useful tool" (E35).Superseding repetitive and deferring procurement procedures, they allow to "activate the services with short notice" (E21) and with "less administrative [effort]" (E35).Finally, experts recommend exchanging best practices and templates with other humanitarian organizations and engaging professional external support, for example, experts for the tender process or contract design.
4.5 Rank 5: set up and maintain a plan for the engagement of logistics service providers (P02) Outsourcing does not only require appropriate preparation (see Section 4.4), but also sufficient and adequate planning.It starts with aligning the outsourcing approach to the overall logistics strategy and developing an exact understanding of service needs and requirements.Agencies should "identify gaps in the infrastructure" (E26) and determine how LSPs can best help to bridge them.Conditions in disaster relief operations often change considerably within short time.As one consequence, agencies should budget for the handling of challenges during outsourcing engagements, as those are almost inevitable in the light of rapidly changing circumstances.As another consequence, organizations need to prepare for these eventualities through scenario planning."If you have the plan in the drawer and [even if it] is 6 months old […], at least you have something, you have a start" (E5).This also implies to source multiple alternative providers for the same service in order to have sufficient fallback options when single providers are not able or willing to provide a service.
Experts recommend complementing these activities by regular risk assessments, which analyze the fit between the outsourcing structure and the potential development of the operational environment.Aid agencies should also involve contracted providers into this process in order to identify "any constraints from their side" (E25).For example, they should "ask LSPs to share their contingency plans for […] any unwanted situation" (E22), such as alternative routes of transportation.Experts even recommend agencies to test these backup plans together with providers, for instance, by routing single shipments on different paths, whenever time allows: "If it's working, we know that we have a plan B. If it's not working, […] we have to search for something else" (E10).In addition, organizations should agree with existing partners upon prices for potential future service requirements, as long as the principles of fair and transparent procurement are not violated.
Finally, planning enables aid agencies to negotiate with providers based on consolidated service demands, for example, aggregated across regions, and to coordinate service purchases with other aid agencies.Coordination, at worst, reduces competition for limited capacities.At best, it offers the opportunity to tender services jointly with other agencies increasing an organization's bargaining power vis-à-vis service providers.
4.6 Rank 6: take actions to ensure the selection of the right provider for the right task (P05) Experts consider it important to invest time and efforts in the selection of the right service provider, since "without the right provider, [the] right service cannot be ensured" (E19).

Outsourcing logistics in disaster relief
The definition of "the right provider" is, however, very context-dependent, since no provider can perform best in all situations.Aid agencies should assess, which type of service provider is best suited under which circumstances, and, instead of just selecting one provider, source a "suit of providers which you can call upon" (E13).This allows the flexibility of working with the best fitting provider in each situation (see Section 4.5).
Experts provided specific recommendations for the selection of the right provider for the right task.Organizations should approach as many LSPs as possible with a service request and allow providers sufficient time to respond.In the case of open tenders, good media support should be secured, for example, through publications in well-established newspapers.When choosing between different service providers, information becomes paramount.However, this can be a major issue in disaster relief operations: "The main challenge is, actually, that we don't have information to make the right selection" (E8).Accordingly, experts recommend various activities to fill this void of information.Particularly less known providers should be asked to provide customer references together with their proposals.These documents offer the possibility to obtain information about the provider directly from other organizations.In general, aid agencies should make use of their network.By reaching out to other aid agencies, local authorities, donors or product suppliers, they can benefit from a "solid base of […] sources that are already verified by somebody else" (E25) and avoid negative surprises.Experts emphasize the importance of the Logistics Cluster in this regard, because "one of [its] key mandates is information management" (E8).
Once provider proposals have been submitted, requesting bid presentations and questioning LSP representatives in person are considered useful by experts.These activities help to identify the true logistics capability of the provider.Measures such as site visits and physical checks of the provider's assets have the same objective.Without activities of this sort, agencies face the risk of providers not being able to perform the service at the promised service level and price.Experts explain that "[w]ritten proposals contain false information just to qualify [for] the minimum requirements" (E22) and that providers "on purpose underrate their offers to beat their competition" (E25) or because they "do not know their own costs" (E7).The capability of providers can also be screened by first testing the contractors under less critical conditions before involving them in important operations.Another very useful activity is, according to experts, the execution of background checks on the provider, for example, regarding its independence, legal compliance or financial health.This is particularly critical at the field level, where providers with dubious practices or links to military groups can not only harm an agency's reputation, but also endanger its employees.

Rank 7: avoid unnecessary liabilities and risks (P08)
The conflicting goals of service providers and the unpredictability of disaster relief operations inevitably lead to risks for aid agencies engaged in outsourcing.Therefore, panelists regard it as important to avoid unnecessary liabilities and risks.While longer-term contracts help to build relationships with providers (see Section 4.9), experts consider them contraindicated in a number of situations.For example, in the beginning of a relief operation, "when suppliers […] raise the prices and it's expected that a few months after [wards] the market will start settling down" (E25).Also, in "volatile economic situations, especially [with] rapidly fluctuating exchange rate[s]" (E31) or when "working for the first time with a service provider" (E15), contracts should be set up as short as possible.However, even in these cases, organizations should "make sure that it is more than just a one-time commercial contract, so that the service provider can be used […] in the future" (E2).
If longer-term contracts are set up, organizations should ensure a sufficient level of flexibility by avoiding any kind of volume commitments or exclusivity clauses.Terms of the latter type might require agencies to treat a LSP as "sole source provider unless there is a 452 JHLSCM 9,3 capacity issue" (E28).Such agreements are considered risky, because "capacity is different to capability" (E28) and "complementarity of coverage by multiple providers" (E38) is highly important (see Section 4.5).Panelists, furthermore, recommend specifying sufficient provisions for contract termination, for example, due to funding issues or the end of a mission, and to agree upon future price adjustments according to prevailing market prices.
In general, oral communication should always be followed-up in written to have proof of the conversation.This is especially important in situations of high urgency, when organizations might be forced to "hire someone […] without […] a formal contract" (E16).In such cases, experts recommend to, at least, request a written commitment from the provider to adhere to the agreed service fees.If a formal contract is set up, it should either be reviewed by a legal expert or be based on a standard template released by the legal department.
Finally, in particular at the field level, it is important to avoid communicating sensitive data to providers, for example, beneficiary lists or budget details, as this could cause severe security issues.Additionally, agencies should avoid too close personal relationships with providers and always "remain [in] professional distance" (E8).In some cultural areas, "for a private company […] it's normal to give incentives" (E17) and organizations should make sure to avoid any impression of favoritism.
4.8 Rank 8: run a formal process for performance evaluation (P09) Bad performance of service providers can be a serious problem for aid agencies, affecting its reputation and the well-being of beneficiaries."The quality of services may vary not only [within] the country context, but also [for] the same supplier over time" (E16).Therefore, panelists deem it important to run a formal process for evaluating the provider's performance.According to the experts, there are two main dimensions of performance evaluation: continuous quality control and regular performance reviews.Especially "when working with local 3PL firms, quality control is a major issue" (E6).Agencies should closely monitor and track the execution of activities by the service provider, always involving those employees who have the best visibility of the service.According to Expert 10, this is "something that we are daily doing by calling our colleagues."Likewise, inspections and audits on service quality and compliance are advisable, although they should be applied carefully as they can "often cause unnecessary delays" (E22).
Besides performing continuous quality control, organizations should conduct a formal review of performance, taking place at specific milestones or in fixed intervals.On these occasions, agencies should use key performance indicators (KPIs) to evaluate the provider's performance against benchmarks and contractual agreements.However, organizations "need to be careful with KPIs, [and] the key is to use only a few" (E28).Furthermore, they need to account for the fact that disaster relief "is not normal, it's generally chaos; […] your normal business-as-usual KPIs just don't apply and shouldn't apply" (E21).Accordingly, indicators which measure the humanitarian performance of the provider are required.They should cover both quantitative factors, such as timeliness or cost, and qualitative aspects, such as transparency or agility.It can also be helpful to involve the service provider into the definition of the performance metrics as this facilitates an "alignment of expectations between the parties" (E21).Aid agencies should, moreover, maintain a record of performance for each provider, which is fed into a service provider database and stored across operations and engagements (see Section 4.4).This allows to use the information not only for the single performance review, but also "as basis of continuous improvements" or as a "negotiation tool" (E16) for future negotiations.Finally, despite the need for performance management, experts emphasize the importance of "keep[ing] the focus on impact of programming rather than donor priorities [or] shipping metrics" (E29).In the end, the "main performance indicator [is] successful delivery of relief items to the beneficiaries for life saving operations" (E2).

Outsourcing logistics in disaster relief
4.9 Rank 9: build mutual understanding and trust with provider (P10) According to the experts, high levels of trust and understanding between an aid agency and its service providers bring significant advantages.For example, providers are more likely to go beyond their contractual obligations in critical situations when they perceive the engagement to be a trustful partnership: "the relationship is everything" (E28).Consequently, panelists consider it important to build mutual understanding and trust with providers.
Building trust starts during the tender process and "the best tendering approach should involve establishing a longer-term partnership" (E7).Therefore, organizations should avoid too much variation in their supplier base: "We only have about 4 service providers […] that we work with on a consistent basis and have long standing relationship[s] with all of them.
[…] [T]his aspect of close relationships helps us work together towards a common, missionoriented goal" (E29).However, this approach "really clashes against this fair, open, transparent scheme" (E5) which is requested by donors and limits the options for building relationships: "Even though we had a fruitful relationship [with a provider], we still have to treat them like everybody else" (E5).Helpful and still compliant measures are personal invitations to public tenders for preferred partners, in order to ensure their participation, and feedback to loosing tenderers, in order to increase transparency and build rapport.
When working for the first time with a provider, aid agencies should explain the specifics of the humanitarian context to the provider's staff and define relationship managers on both sides, because "building trust in the beginning […] often depends on individuals in [sic.]both the company side and the NGO side" (E14).In this regard, it is also advisable to have selected staff of the provider working within the aid agency for some time, or vice versa.This does not only "bring […] special expertise into the NGO" (E3), but also helps to gain "a better understanding [of] the internal dynamic procedures of the partner" (E35).Due to the risks of close personal relationships (see Section 4.7), panelists deem this measure more suitable at the headquarter level.Experts, furthermore, recommend setting up contracts with longer durations as long as the circumstances allow it and sufficient provisions for termination and adjustments are included (see Section 4.2).In particular for local services, they should review the contract jointly with the LSP, because "in many cases […] suppliers do not even read the contract, they just sign any document to get the contract" (E19).
Overall, experts emphasize that building trust is a "two-way street" (E25).Therefore, agencies should always keep their own commitments, for example, with respect to payments.Moreover, they should be flexible if changes to agreed terms are required.They should "understand the situation of the service provider" and "have [a] fair manner of treating the partnership, […] not necessarily sticking 100% to the contract" (E8).Aid agencies should, additionally, share projections on future service demands with selected providers to signal their interest in long-term relationships.
4.10 Rank 10: work with provider as an integrated team (P12) "When you can engage the service provider as a core team member, not as a general support entity, [it] offers the most" (E22).Consequently, panelists consider it important to work with providers as an integrated team.Aid agencies should, first of all, define contact persons and corresponding backups, which are available for phone calls during 24 hours a day.Moreover, they should target to work face to face with the service provider whenever possible, because "as soon as you are working face-to-face for some time, you build trust, you build understanding, and you have [a] better relationship" (E14).However, at the same time, they should take measures to avoid any impression of too close personal relationships or favoritism, especially at the field level (see Section 4.7).
Once the tender process has been finalized, experts recommend involving the provider in the logistics planning of an operation (see Section 4.5).Furthermore, aid agencies should 454 JHLSCM 9,3 proactively share all relevant information with the provider and, in general, communicate frequently, honestly and transparently.In time-critical operations such as disaster relief "the common level of agreement is vital" (E22) and "regular communication […] may minimize the ambiguity and speed up the process" (E31).Additionally, organizations should cooperate with the provider to find joint solutions for challenges and even "help it to best being able to fulfil its contract in a safe and practical manner" (E27).Treating the provider as a team member, according to panelists, also requires investing in the development of its capabilities.Therefore, organizations should not only provide continuous performance feedback to providers and regularly discuss "lessons learnt", but also involve the vendor's staff in internal trainings on standard operating procedures and best practices: "for example, we train the drivers in how to drive economically and ecologically" (E2).
Finally, information technology should be leveraged to enable smooth cooperation.Data should be exchanged electronically between IT systems, in best case through electronic data interchange (EDI).If regular information exchange is required, for example, in the case of warehousing services, experts consider it useful to give the provider access to the aid agency's IT system, as long as the system allows to restrict the provider's access to the required modules and functionalities.

Rank 11: design engagement as win-win situation (P11)
In general, panelists agree that the engagement should provide sufficient benefits to both sides, because "partnering with a LSP offers much more [when both] NGO and LSP […] benefit" (E28).If "both the client and the provider are satisfied with the agreement, [they] will do all efforts to ensure its success" (E23).Accordingly, participants deem it important to design the engagement as win-win situation.However, a lot of controversy exists with respect to the best approach for achieving it.A key reason might be the "historic relationship the humanitarian world was having with the private sector, which has been not too trustful and [has] been very much looking at the private sector [as] someone who just wanted the profit and just wanted to cheat" (E8).
There are still sceptical opinions: "commercial profit is by all means enormous" (E19) and "when emergency situation[s] occur […] genuine competition seem[s] to disappear" (E21).Nevertheless, panelists consider it useful to agree upon fair service fees which allow for adequate provider profits, because "we engage with the commercial world which needs to generate profit" (E21).Likewise, experts recommend both the use of cost-plus contracts, which link service fees to the actual costs of providers, and the specification of price adjustments according to official cost indices.These are not only fair approaches, which safeguard providers against increasing costs, but also avoid that LSPs charge premium fees to hedge risks or that a provider "reduces the quality of the service […], because he has to get that [increasing] cost from somewhere" (E16).While experts agree on the advantages of index-based price adjustments, they highlight that the approach is "not easy to manage" as it's "not very popular when the fuel price goes down and you have to [lower] the rates" (E8).
Panelists also recommend acknowledging a good performance of providers by providing a letter of reference, which they can use in future tenders.Some experts suggest making faster payments, directly extending contracts in case of satisfactory provider performance and guaranteeing fixed minimum payments to providers, independent of the transaction volume.These activities are, however, not considered useful by other panelists.Similarly, a lot of controversy exists with respect to performance-based service fees, which include a bonus for performances above target.Some think that "offering more money for better service never works" (E22) and that it would be very difficult to justify to donors "that we have paid somebody more than what they asked" (E25).Others argue that "imposing penalties for suppliers is default, but giving incentives is not" (E19) and actually "carrot and stick both should be [t]here" (E19).Also, letting the provider benefit from the humanitarian Outsourcing logistics in disaster relief image through publicity is seen controversially.While it can help to get better and cheaper services, it might not be possible due to donor requirements or security issues.Some argue that, if at all, it should only be granted in case of free-of-charge services.
4.12 Rank 12: set up financial incentives to align goals of service provider (P07) The importance of setting up financial incentives is seen controversially by experts.While some consider it not important or only somewhat important, others deem it important or even very important.This difference in judgments appears to be due to the complexity and context-specificity of defining appropriate financial incentives.
Without controversy, experts recommend defining adequate documentation by service providers as precondition for payments.Failure in this regard, for example, missing waybills, can lead to serious conflicts with donors.Organizations should also avoid payments in advance of the service, wherever possible.However, such payments may be required, because "in certain cultures, in certain places […] you have to deposit the amount" (E5) and an "advance payment is the only way to get a service" (E8).Likewise, such payments might be necessary if the provider is "just a small company, with little cash flow […] [and] cannot start without this payment" (E35).Therefore, aid agencies should use their best judgment and accept that this is a "risk you have to take […] [and]

explain […] to the donors" (E5).
There are also opposing views regarding the use of open-book contracts, in which a provider reveals its actual costs to the aid agency.Some experts have experienced them to work extremely well, because they offer a high transparency and reduce the risk of being cheated.Other experts emphasize, though, that "depending on the culture […], [open-book contracts] may be synonymous to no trust" (E16) and can negatively affect the partnership (see Section 4.9).Furthermore, facilitation payments or bribes can cause challenges for open-book contracts: "if you need to get things through certain areas, customs, there will be certain fees […] that we really can't substantiate […] because it's an ineligible cost [for donors]" (E5).
As another financial incentive, experts suggest guaranteeing contract extensions in case of satisfactory provider performance, even though an agreement on the definition of "satisfactory" can be difficult to achieve.Finally, panelists recommend performance-based service fees, i.e. fees that are reduced if performance is below a pre-agreed target.Such penalties can, for example, be defined for late deliveries.However, "you can only apply [penalties] to factors within the 3PL control, which may be limited in a humanitarian context" (E7).In general, penalties conflict with the objective of building mutual trust and understanding (see Section 4.9).Therefore, penalty terms "need to be fair and realistic" (E21) and organizations should carefully weigh their application: "penalties are often […] part of the contract, but […] rarely applied […].Best is to be able to work with the supplier" (E24).

Discussion
In this section, we reflect on the results in view of relevant literature on commercial and humanitarian logistics, which has been presented in Section 2. According to the expert panel, the most critical practices for ensuring the success of outsourcing logistics are balancing compliance and efficiency in the procurement process (P03), using detailed written contracts (P06), and issuing detailed requests for proposal or quotation (P04).All of them are considered very important (mean rating ⩾ 3.50).
Practice P03 is not only considered the most important one, but also differs in another regard from the remaining practices: it was the only practice which we could not map during the process of deductive coding, at least to some extent, to success factors from commercial logistics literature (see Table II).The practice seems to be irrelevant in the commercial context (Selviaridis and Spring, 2007).While commercial enterprises are independent of donations, donors play a significant role in the humanitarian environment (Wakolbinger and Toyasaki, 2018).Their requirements have to be considered in many ways when 456 JHLSCM 9,3 outsourcing logistics, for example, with respect to the applicable procurement procedures, remuneration forms or payment terms.They impose "a major constraint on our ability and as organizations" (E5).If their requirements are violated, donors can withdraw their funding causing serious financial problems for agencies.As, simultaneously, time pressure and speed play an outstanding role in emergency operations (Van Wassenhove, 2006), it appears plausible that experts consider Practice P03 a very important addition to the elements from the commercial sector, and, in fact, the most important of the 12 best practices.Quite contrary to the experts' view, the importance of compliance requirements for outsourcing has not been stressed a lot by literature on humanitarian logistics.In general, academic literature has not investigated sufficiently the implications of donors on operational decisions (Burkart et al., 2016).Our results, similarly to Bealt et al. (2016), suggest future research in this regard.
Different than for Practice P03, the relevance of the two other very important practices, P06 and P04, is also highlighted by the existing literature.It is recognized that commercial enterprises should align expectations, provide well-defined requirements and be thorough in the negotiation of a formal contract, which defines clear ground rules and includes provisions for adjustments and exit (Hofenk et al., 2011;Selviaridis et al., 2008;van Laarhoven et al., 2000;Lambert et al., 1999;Frankel et al., 1996;Tate, 1996;Leahy et al., 1995).

Exemplary references
for best practices in literature on commercial logistics

Outsourcing logistics in disaster relief
Also humanitarian organizations should set-up formal contracts, ideally in advance of disasters, and be clear in their requirements toward service providers (Baharmand et al., 2017;Cottam et al., 2004;Bealt et al., 2016).As humanitarian organizations and LSPs have inherently diverging objectives and motivations (Nurmala et al., 2017), aligning expectations is potentially even more important than in the commercial context.A detailed service request and a detailed contract, as a manual of cooperation (Hauptmann, 2007), can help to clarify both requirements and expectations.Our results go beyond these high-level insights and provide clarity on how aid agencies can reflect these objectives within the content of contracts and the process of service requests.Specific requirements of the humanitarian context become obvious on this level of detail.For example, at the time of the service request, not all required information might be available, and both the aid agency and the provider need to show flexibility in negotiating the service.
A major recurring theme in literature on humanitarian logistics is the importance of preparedness for disaster relief ( Jahre et al., 2016).This also applies for the field of outsourcing.By investing in training, relationship building or the setup of service agreements in advance of relief operations, the engagement of the private sector in the response phase can be improved considerably (Bealt et al., 2016;Quero, 2012;Tomasini and Van Wassenhove, 2009;Zyck and Kent, 2014).Our results mirror this perception: according to the expert panel, adequate preparation (P01) and planning (P02) are important practices for ensuring the success of outsourcing.Experts suggest that preparation should cover guidelines, documentation, training, staff development, tools, templates, information gathering and relationship building.While aid agencies recognize the importance of these activities, most of the organizations never or only occasionally make such preparations (Bealt et al., 2016).Donor priorities and a corresponding lack of preparedness funding are certainly among the main reasons for this apparent contradiction ( Jahre et al., 2016).Therefore, low-cost preparedness activities with a high return on investment are of special interest to aid agencies.Our results contain some promising recommendations in this regard, for example, regular assessments of provider markets, which require moderate effort but can lead to significant cost reductions.
The literature, both from the commercial and humanitarian field, also highlights the importance of organizations building close relationships with LSPs and establishing long-term partnerships (Cottam et al., 2004;Maon et al., 2009;Bealt et al., 2016;Nurmala et al., 2017).Especially trust is frequently mentioned as one of the most important preconditions for successful outsourcing relationships (Bealt et al., 2016;Frankel et al., 1996).Our results, on the one hand, confirm this judgment by literature.Experts consider it important (2.50 ⩽ mean rating o 3.50) to build mutual understanding and trust with providers (P10), to work with providers as an integrated team (P12), and to design outsourcing engagements as win-win situations (P11).On the other hand, our results relativize the role of partnerships and trust to a certain extent.Panelists perceive these practices (P10, P11 and P12) considerably less important than, for example, setting up detailed and formal contracts.
The relationship of trust and contracts is certainly a controversial one.It is subject to a continuous academic debate since Macneil (1978) first differentiated classical, neo-classical and relational contracts.It remains often contested if contracts and trust function as complements or substitutes, and which of the two is more important (Boyson et al., 1999;Frankel et al., 1996;Hofenk et al., 2011;Poppo and Zenger, 2002;Spekman and Davis, 2004).Trust is a very complex concept itself, subsuming deterrence-based trust, calculus-based trust and relational trust (Rousseau et al., 1998).Our results suggest that in the humanitarian context contracts are more important, but still need to be complemented by trust: "One thing is to have a good contract, which is the basis and you need to have that, but we need to live beyond the contract in a professional manner" (E5).

JHLSCM 9,3
There are various potential reasons for why experts consider detailed contracts more promising for ensuring the success of outsourcing logistics than establishing trust-based partnerships.The main one appears to be the difficulty of building trust in disaster relief.Hastily formed networks and the lack as well as high turnover of adequately trained staff cause big challenges for developing relationships (Kovács et al., 2009(Kovács et al., , 2012;;Pettit and Beresford, 2009;Tatham and Kovács, 2010).This is specifically true for relationships between aid agencies and commercial actors, which are even more challenging because of differences in culture and working styles (Nurmala et al., 2017).One exemplary consequence is humanitarian practitioners fearing that commercial LSPs might abuse the context of disaster relief for making excessive profits (see Section 4.11) while the financial resources of aid agencies are scarce (Kovács et al., 2009).It appears that aid agencies first need to address the foundations for building trust before they can more comfortably substitute formal contracts by relational ones.For example, they would have to increase their level of communication with partners (Cottam et al., 2004;Tatham and Spens, 2011), create internal awareness for cultural differences (Dowty and Wallace, 2010) and adopt more decentralized decision-making (Baharmand et al., 2017).
It is noteworthy that some of the panelists' recommendations are, at least to a certain extent, conflicting or controversial.For example, experts conform with the literature (Cottam et al., 2004) regarding the fact that close relationships can ease coordination and improve mutual understanding.They also highlight, however, that such relationships can endanger an aid agency's reputation by causing a perception of favoritism.Similarly, experts recognize that long-term contracts can be beneficial for building partnerships (Maon et al., 2009), but counter that they increase an organization's dependency and limit its flexibility.Also with respect to performance-based service fees the panel did not reach a consensus.While some argue that it is important to implement them (Balcik et al., 2010), other consider them counterproductive.As mentioned in Section 3.3, the usefulness of selected activities is certainly influenced by contextual factors, such as the value of the service purchase, size and funding structure of an aid agency, the geographic context of the service (before or after point-of-entry) or the phase of disaster relief.This study's aim was to provide a comprehensive picture on promising activities for organizations engaged in outsourcing.It was an unfortunate, but inherent consequence that we were not able to go into the details of all identified controversies and had to leave some of them unresolved.We suggest that future research provides more clarity regarding the respective matters.

Summary and outlook
This study investigated best practices of aid agencies for ensuring the success of outsourcing logistics to commercial LSPs in disaster relief.It is based on a Delphi study with a panel of 31 experienced practitioners and a complementary focus group with 12 experts.As it is one of the first studies in the field to apply the Delphi method, it also examined the application of the technique to the field of humanitarian logistics.
Our research revealed 12 best practices and a compilation of more than 100 activities for putting these practices into action.Experts consider a proper balance between efficiency and compliance, a detailed contract and a detailed service request the most important practices.The practice of balancing efficiency and compliance is a unique addition of our study to literature from the commercial context and results from the influential role of donors in disaster relief.In contrast to existing literature, building trust and establishing win-win situations have a lower importance for practitioners than most of the other best practices.This apparent contradiction is potentially a consequence of the great challenges for building trust in disaster relief.Apart from providing guidance to aid agencies, the developed ranking of best practices can help to prioritize future academic research, since it reflects the priorities of practitioners.

Outsourcing logistics in disaster relief
The findings have certain limitations and can be extended in several directions.Even though we involved LSPs in the focus group, our results clearly represent the one-sided perspective of aid agencies.While this was our intended focus, future studies in this context could aim to determine best practices for LSPs as well (Heaslip, 2013;Vega and Roussat, 2015).Moreover, our research design did not allow to investigate potential interdependencies between individual practices, which are likely to exist.Future research could deepen the insights in this regard, for example, by applying interpretive structural modeling as illustrated by Qureshi et al. (2007).In general, future studies could further examine specific aspects of the 12 practices.For instance, they could determine criteria for selecting service providers and indicators for evaluating their performance (Baharmand et al., 2017).It would also be important to better understand the role of donors in the context of outsourcing and to identify approaches for further harmonizing their requirements with the operational environment of aid agencies (Burkart et al., 2016).Likewise, future research could provide more clarity with respect to some of the unresolved controversies in our results, for example, how aid agencies should deal with close relationships or performance-based fees.
Finally, we would like to encourage other researchers to explore the use of the Delphi method for their work.We found it to be a promising approach for assessing the perspectives of humanitarian experts in a structured and confirmable way.While specific limitations exist, we strongly believe that these can be addressed by an adequate research design.

Notes
1. Seven participants preferred a phone interview to the online survey.In these cases, we used the online questionnaire as an interview guideline.All interviews took between 45 and 90 min and were transcribed in verbatim form for further analysis.
2. E19 refers to Expert 19 (see Table A1).Direct quotes from experts are provided without modifications.Any adjustments are highlighted using squared brackets.

Outsourcing logistics in disaster relief
Participation (e-Delphi/ focus group (FG))

4. 3
Rank 3: be detailed in the request for proposal or request for quotation (P04) Panelists emphasize that a clear and precise service request is the foundation for a successful outsourcing engagement: "If you do a proper analysis of your needs, and you specify very well what […] you expect from the contractor […], that's […] going to lead to [a] clear contract and contractual obligations" (E25).Based on a clear understanding of their own requirements (see Section 4.5), agencies should explain these in detail to the provider: Notes: Scale: 1not important; 2somewhat important; 3important; 4very important.a No consensus among panelists (IQRW1.0and/or r wg ⩽ 0.3) Appendix 2. List and rating of activities Set short contract duration (e.g.one-time services) in case of high uncertainty (e.g. economic volatility, chaotic situations, at the beginning of an operation)