Survey-based research

3.2.1. The six conditions for effective partnership

The main research tool that was used was survey-based. I applied an existing assessment tool that was developed by public health researchers Dr. Bilodeau and colleagues (2008, 2011) in the context of their study of cross-sectorial partnership in Montreal, QC. The survey aims to assess whether a given public-civil society partnership meets six conditions for an effective partnership. An effective partnership can be assessed based on the dynamics of the process on the one hand (conditions 1-4), and the partnership’s arrangements on the other (conditions 5-6). The conditions are summarized in the table below.

Table 1: Summary of the conditions for effective partnerships²⁵ Conditions for effective

partnership

Description of the condition Number of items 1. Problem-framing All the actors concerned with a common

issue introduce their interests, perspective and position.

2

2. Interest-raising Actors are interested in strategic decisions early in the process, and in playing a role in the partnership.

1

3. Enrolling Actors are brought into a position of negotiation and influence in the decision making process and accept temporary and 6. Co-constructing Actors resolve controversies that divide them

and integrate their work around collective solutions.

6

25 The items for each condition are available in annex 2. To see the full results of each item and conditions, see the appendix.

25 Each condition is divided into a number of statements. One descriptive statement is associated with conditions 2 and 3, two statements with condition 1, four statements with condition 4, and 5, and six statements for condition 6. In total, the six conditions for an effective partnership are presented into 18 items, which are presented in annex 2 of this thesis. These items are not presented as questions, but as statements. For each of these statements, respondents can choose from three formulations that best describe their experience, each formulation corresponding to a degree of fulfillment (fully, partially or not at all).

The survey was sent out in French. Once the responses were received, I compiled each of the 18 items based on their degree of accomplishment. I then compiled the degree of accomplishment of each item into the six summary conditions and classified the items and conditions based on whether they were completely fulfilled, partially fulfilled or not at all.

Based on previous studies, I judged that an item or condition was fulfilled if more than 70% of respondents identified them as such. On the other hand, we considered an item or condition was partially fulfilled if half of the respondents considered them so.

I classified items that received fewer answers, indicating which items respondents were unable to formulate an opinion on. For the conditions that were not fully met, I conducted a sectorial breakdown based on whether respondents were from civil society, government agencies or others. I met with Dr.

Bilodeau, one of the key authors behind this tool, on two occasions to discuss and analyze the results.

3.2.2. Modeling the partnership network

Actor-Network Theory echoes network analysis and visualization pertaining to the discipline of Social Network Analysis (SNA). “Social network analysis provides a means with which to identify and assess the health of strategically important networks within an organization. By making visible these otherwise “invisible” patterns of interaction, it becomes possible to work with important groups to facilitate effective collaboration.” (Cross et al., 2002, p. 41). In the context of the SAM 2014-2016 action plan, I wished to capture the changes in the network in the SAM 2014-2016 by modelling a

“before” and an “after” the launch of the action plan.

The second section of the survey consisted of a grid-based response tool where respondents could identify whether they collaborate with others actors in the network, and if so since when. Respondents could choose from a series of options that were designed based on two axes, one vertical and one horizontal. The vertical axis indicates the name of each member of the SAM partner committee (n=36).

The horizontal axis presents longitudinal options with the SAM action plan (2014-2016) as a reference point, a before (pre-2014) and after (2017-2020). For the 2014-2016, respondents could choose from

26 two options, that is, whether they started collaborating with another actor thanks to (“via”) the SAM, or outside of the SAM (“without”).

In contrast with the conditions for an effective partnership, which was developed by Bilodeau and colleagues (2008), I designed the section of the network modelling section of the survey. I ran into a methodological dilemma that is worth presenting here. In its original formulation, the survey was designed in such a way as to capture the dynamic of the social relationships, ranging from information sharing to working on one or more projects together. Bilodeau and colleagues have previously used a typology in the shape of a continuum, ranging from information exchange and project conception, to planning and execution. However, being in an evaluation context with a focus on showing results, subsequently led me to look at whether the SAM partnership facilitated new relationships. This led me to privilege a four-period longitudinal analysis.

In order to visualize the network, I first had to organize the data in such a way that the Gephi open source software, a social network visualization tool, could process it. The first step was to create a spreadsheet that would eventually identify the “nodes” in the subsequent network graphs. I assigned to each organization, an identification number as well as two sectorial attributes (a) social, environmental, economic, and (b) public network, civil society, philanthropic, public-community, private. I merged respondents originating from the same organization, which would eventually create a bias, as organizations with one or more staff involved in the SAM partnership appear more central in the network.

Once I had an identification number for each organization, I developed a series of spreadsheets with the “edges” (or connections) based on each corresponding time period (before 2014, 2014-2016, after 2017). All the connections identified by each respondent were then compiled and organized under two columns, one designated as “source” and the other as “target”, before being modeled in the software. In the case of the 2014-2016 database, each connection was associated with one of the two following attributes based on the survey answers (“via” the SAM, “without” the SAM). This characterization was meant to make observation whether partners recognized the Action Plan as contributing in creating new connections in the network.

Once these initial steps were completed, the spreadsheets were imported into the software. In order to read the graphs, we formatted the layout (Force Atlas, Repulsion Strength), and ranked nodes according to social network metrics, such as between-ness and degree centrality²⁶. In the end, we designed twelve network graphs based on time period and on sectorial attributes, available in Annexes

26 Betweenness centrality measure the centrality of a node. Degree centrality measures the number of connections of a given node.

27 3 to 7. I generated other network metrics available through the software (connected components, graph density, average path length) (Annex 10).

3.2.3. Sampling and responses to the survey

The survey was sent to 68 of the partners on the mailing list of the SAM partners committee (Annex 9) in mid-March 2016. Three general reminders and individual reminders were sent. The online survey was left active for six weeks, until early May. The survey was self-administered and filled out online by participants. The survey introduction included instructions for its completion, the estimated time required to fill it out (20 min.) and the protocol on ethics²⁷.

The final version of the survey included 33 questions, grouped in the following sections:

1. Introduction (8 questions)

2. Graphs of the partner networks (social network analysis) and of the impact of the SAM on the development of the network (2 questions)

3. Assessment of the six conditions for an effective partnership (18 questions) 4. Open questions (5 questions)

The sample was made up of the SAM coordination email list to the SAM partner committee, which is made up of thirty-six organizations. The selection criteria for this mailing list were therefore developed beforehand by the SAM, and correspond to the partner committee members, project leaders or project participant-experts. In order to capture the information connections and the professional aspect of the SAM network, I designed the survey in such a way that it was answered individually, rather than at the organizational level.

In this sample, 33 answered for a response rate of 48.5%.

Respondents’ participation levels in the SAM:

• Coordination committee (18.7%)

• Partner committee (59.4%)

• Participants (25%) (i.e., respondents that are not officially members of either committee)

Respondents’ activity sector

• Public sector (40.6%)

• Civil society authority or organization (40.6%)

• Public-community authority; philanthropists or public-private partnerships; universities and research settings (6.3% each)

Level of respondents’ involvement:

27 https://goo.gl/1bqhqR

28

• Neighbourhood or borough (17.6%)

• Municipality (23.5%)

• Agglomeration (61.8%)

• Metropolitan community of Montreal (11.8%)

• Province of Quebec (26.5%)

• Other (8.8%)

The sample is considered to be sufficiently representative of the partners’ different levels, scales and sectors of involvement. One major cause of bias in the social network graphs was that several respondents belonged to the same organization. Further, because I provided the respondent with a limited number of actors to identify, and given the large scope of interpretation of what it means to

“collaborate” with a partner organization, a high number of ties were recorded and the network graphs are extremely dense. This makes the network graphs difficult to interpret and analyze. I extracted a complementary graph from the evaluation report I authored: this network graph, available in Annex 11, was developed as part of a survey to project leaders, asking them to identify the three most involved partners in the actions funded in the SAM Action Plan.